Implementing Goal Mama: Barriers and Facilitators to Introducing Mobile Health Technology in a Public Health Nurse Home-Visiting Program

Abstract

The present study explores barriers and facilitators experienced by public health nurses introducing a mobile health technology platform (Goal Mama) to the Nurse-Family Partnership home-visiting program. Goal Mama is a HIPAA-compliant goal-coaching and visit preparation platform that clients and nurses use together to set and track goals. Forty-two nurses across five sites, including urban, suburban, and rural communities, piloted the platform with clients for 6 months. The mixed method, QUAL+quan pilot evaluation focused on deeply understanding the implementation process. Data were analyzed via iterative content analysis and multivariate regression analysis, and triangulated to identify salient findings. Over 6 months of use participants identified critical areas for product and implementation improvement, but still viewed the platform favorably. Key opportunities for improving sustained use revolved around supporting the technological and programmatic integration needed to lower key barriers and further facilitate implementation.

Introduction

Home-visiting programs are promoted and supported by the U.S. federal government’s Maternal and Child Health Bureau as an evidence-based strategy that can significantly change the trajectory of women and children living in poverty. The Nurse-Family Partnership^® (NFP), one of the oldest and most-researched programs, pairs a young, low-income first-time mother with a public health nurse (PHN) who provides regular home visits during pregnancy and through the child’s second birthday. A woman is eligible to participate in NFP if she (1) is pregnant and less than 28 weeks into pregnancy, (2) has had no previous live births, and (3) meets certain low-income criteria. NFP PHNs provide personal guidance and expert health care advice to guide mothers through the challenges of pregnancy and early parenthood, counseling their clients on everything from prenatal health and child development milestones to goal setting and stress management. PHNs carry a caseload of up to 30 mothers and are supported by a nurse supervisor (Nurse-Family Partnership, 2017).

The NFP program is implemented by approximately 2,400 baccalaureate-educated registered nurses nationwide through a network of approximately 270 community-based implementation partners in 40 states, the US Virgin Islands and several Tribal communities that span rural, urban and suburban settings (Nurse-Family Partnership, 2021). The program currently serves over 38,000 families each day, and the NFP model has been proven effective through four decades of research and evidence gathered in three separate randomized, clinical trials in Elmira, NY, Memphis, TN, and Denver, CO. Evidence from the original trials as well as continued rigorous evaluation has demonstrated significant and durable improvements in health, social, economic and life course outcomes for both mothers and children (Olds, 2002; Olds et al., 1986, 1994, 1997, 1998, 2002, 2004).

Improving and scaling even the most impactful programs requires innovation, and mobile technology is one promising way to help amplify an efficacious program’s impact and scale. A total of 96% of young adults in the US are using smart mobile devices (Pew Research Center, 2019), and in one national sample of 1,604 mobile phone users, more than half (58%) had downloaded a health-specific application (app) (Krebs & Duncan, 2015). In low-income and traditionally underserved populations, mHealth apps that are sensitive to user needs have the ability to catapult access to healthcare (Liu et al., 2020). Apps have the potential to supplement the NFP model by increasing access to care enhancing communication between mothers and PHNs.

In the maternal health space in particular, studies show that women were already increasingly accessing digital information to supplement dynamic elements of their maternal health care (i.e., prenatal care, baby care, relationship issues) (Wexler et al., 2020). Internationally, studies have found that 97% of pregnant women use the internet to get information on pregnancy, 94% use the information obtained digitally to augment the guidance provided by their health care provider, and 83% take the information they discover online into consideration when making health care decisions (Bert et al., 2013; Lagan et al, 2010). Use of apps by patients is ubiquitous and is gaining traction by providers and health systems (Bert et al., 2016; Connor et al., 2018; Lee & Moon, 2016; Tripp et al., 2014).

Prior to 2018, NFP had not piloted or adopted technological innovations, but knew, based on focus data from program participants and anecdotal PHN experience, that much of the population served were digital natives (The Idea Marketing, 2017, 2018). The introduction of a digital product was aligned with the NFP model’s commitment to serve clients in a way that aligns with how they are most likely to seek information and services. Program leadership was aware that to remain relevant to the population NFP is intended for, technology could be introduced as part of a focus on continuous improvement and innovation, while maintaining the evidence-based model of care.

While clinical evidence for app use grows, the challenges to endorsement and adoption by health systems and organizations remain. Provider acceptability and usability top the list of challenges (Boudreaux et al., 2014; IQVIA, 2017). However, these barriers are not insurmountable, and system-wide promotion of digital technology can be a driver of implementation (Doshi et al., 2020; Keesara et al., 2020). When service providers are able to adopt technology to augment the care they deliver, evidence suggests that this enhances opportunities for patient engagement, patient-provider relationship, patient satisfaction, and improved outcomes (Cueto et al., 2019; Daly et al., 2018; Frazer et al., 2015; Leventhal et al., 2012; Overdijkink et al., 2018; Rief et al., 2017).

This study explores the barriers and facilitators experienced by PHNs when introducing a mobile goal coaching and home-visit preparation platform (Goal Mama) to the NFP home-visiting program in 2018. As part of implementing new technology to the NFP model, we specifically wanted to explore key domains of acceptability, feasibility, and utilization to inform intervention improvement and dissemination at scale. We aimed to address the following implementation-focused research questions:

1. What is needed to integrate Goal Mama at NFP sites?

2. What changes to Goal Mama might increase the acceptability, appeal and impact of the platform?

3. How do PHNs rate the acceptability and feasibility of Goal Mama over the course of implementation?

Intervention

The Goal Mama platform was developed for NFP, in collaboration with Hopelab and Ayogo Health Inc (henceforth, Ayogo). Hopelab is a non-profit social innovation lab that creates behavior-change technology to improve adolescent and young adult health, and Ayogo is an evidence-based digital health company that develops HIPAA-compliant apps for patient engagement. Goal Mama is a product that NFP clients and their PHN home-visitors can use together to set and track personal goals. The platform includes an app for the client, as well as a dashboard for the PHN.

A primary function of the app is the goal setting and tracking feature, which was developed based on foundational evidence on the stages of goal setting and achievement (Gollwitzer & Sheeran, 2006; Locke & Latham, 2002; Locke et al., 1988; Sieloff & Frey, 2015; Stock & Cervone, 1990). Goal setting is a core part of the NFP model, guided by a client-centered principle that even small changes are important. To that end, opportunities for goal setting, progress, and achievement are addressed in every visit, and considered a key driver of behavior change around the primary outcomes of positive pregnancy outcome, positive child growth and development, and parental economic self-sufficiency. Goal Mama elicited goals related to school completion, job attainment, child development, and health behaviors, such as increased physical activity, improved diet, greater hydration and smoking cessation.

The app also includes features designed to help clients and PHNs better prepare for their visits together, such as functionality for clients to let their PHNs know what they would like to discuss at their upcoming visits, and scheduling functionality that reminds clients about upcoming appointments. The app includes additional features that were designed based on clients’ and PHNs’ needs, and which draw on the available evidence base for creating effective health engagement, behavior change technology, and mobile apps (see Supplemental Materials for screenshots and detailed description of feature-set).

While clients have access to a client-facing app, the companion dashboard allows PHNs to view clients’ progress on their goals and their activity in the other app features. The dashboard is where PHNs can prepare for upcoming visits by viewing what discussion topics a participant has selected in their app. The dashboard also provides a mechanism for PHNs to send encouraging messages to their clients in between visits, which show up in the clients’ app. The transition from paper-based goal-setting and visit preparation to technology-augmented, interactive processes allows PHN’s to support their clients based on real-time data about their goals, dreams, and top priorities for upcoming visits. Additionally, the organization is able to track the number and type of goals that clients set and accomplish over the course of their program participation.

Methods

Approach/Design

This was a 6-month, mixed-method, QUAL+quan pilot study that used The Consolidated Framework for Implementation Research (CFIR) to explore facilitators and barriers to implementing the Goal Mama platform within the NFP program. CFIR was used to identify relevant domains and constructs to guide our specific research questions, instrument development, and analysis (See Table 1). While several implementation theories exist, CFIR was developed to consolidate existing theoretical frameworks into both an explanatory framework and process for systematically exploring barriers and facilitators when introducing an innovation (Breimaier et al., 2015). It is made up of five domains (intervention characteristics, outer setting, inner setting, characteristics of individuals, and process) and 39 constructs and sub-constructs (Damschroder et al., 2009). The constructs can be used to explore whether the implementation of an innovation may or may not succeed, identify the barriers and facilitators that arise, as well as strategies to address these factors. As shown in Table 1, our team used the five CFIR domains to guide the development of our research questions, select the relevant constructs to build our data collection instruments, and analyze the data.

Table 1.

Descriptions of Applied CFIR Domains, Constructs and Emergent Themes.

CFIR	Construct description	Emergent theme	Theme description
Intervention characteristics
Trialability	The ability to test the intervention on a small scale in the organization.	Integration	Integration of Goal Mama into existing NFP programs, workflows, or processes.
		Nurse capacity	The logistical or technological ability of nurse participants to adopt and use the app and dashboard with their clients as intended.
Outer setting
Client needs and resources	The extent to which patient needs, as well as barriers and facilitators to meet those needs, are accurately known and prioritized by the organization.	Perceived alignment with client needs	The extent to which the app/dashboard effectively addresses and meets the client needs it was designed to meet, or additional unanticipated needs.
		Perceived client acceptability and feasibility	The perception among nurse participants that the app/dashboard is agreeable or satisfactory to clients.
Inner setting
Compatibility	The degree of tangible fit between meaning and values attached to the intervention by involved individuals, how those align with individuals’ own norms, values, and perceived risks and needs, and how the intervention fits with existing workflows and systems.	Alignment with nurse needs	The extent to which the app/dashboard meets nurse needs in terms of content (appropriateness), feasibility, functionality, and utility.
Implementation climate	The absorptive capacity for change, shared receptivity of involved individuals to an intervention, and the extent to which use of that intervention will be rewarded, supported, and expected within their organization.	Integration	Integration of Goal Mama into existing NFP programs, workflows, or processes.
Access to knowledge and information	Ease of access to digestible information and knowledge about the intervention and how to incorporate it into work tasks.	Nurse learning	Needs or insights related to learning (initial or follow-up) on the part of the nurses when it comes to either the Goal Mama app or the Nurse Dashboard.
Characteristics of individuals
Knowledge and beliefs about the intervention	Individuals’ attitudes toward and value placed on the intervention as well as familiarity with facts, truths, and principles related to the intervention.	Perceived client acceptability and feasibility	The perception among nurse participants that the app/dashboard is agreeable or satisfactory to clients.
Self-efficacy	Individual belief in their own capabilities to execute courses of action to achieve implementation goals.	Confidence in app use	Level of nurse participant confidence in their own ability to use the app/dashboard, introduce the app to clients, and support clients in ongoing use/troubleshooting.
Implementation process
Implementation leaders	Individuals from within the organization who have been formally appointed with responsibility for implementing an intervention as coordinator, project manager, team leader, or other similar role.	Role of the Nurse Supervisor	Content of the role, active or passive, that Nurse Supervisors play in the implementation of Goal Mama during pilot.

The QUAL+qual design was determined based on the stage of implementation (adoption) and the ability to collect data to understand both processes and outcomes (Aarons et al., 2012; Palinkas et al., 2011, 2019). Qualitative data was collected through on-site observations, focus groups, and open-ended survey responses, while quantitative data was collected via closed-ended survey responses. Observational data were collected at each site at the study outset as PHNs were being trained on how to use the platform. We conducted nine telephone-based focus groups over the course of the 6-month study period. Focus groups ranged from 6 to 12 participants and elicited specific feedback on PHNs’ experiences implementing Goal Mama into their practice.

The collection of qualitative data throughout the course of the pilot allowed us to explore the implementation process in depth, capturing the details of how PHNs experienced the adoption of the Goal Mama platform. Quantitative survey data collected at two time points was used to look at specific implementation-related outcomes of interest. As is typical in mixed-method implementation research, we brought the data together during analysis to provide a depth and breadth of understanding of how the process and outcomes data built on each other (Palinkas, 2014).

Setting and Ethical Oversight

The Goal Mama platform was piloted at five of the approximately 270 sites implementing the NFP model across the country. The NFP model is embedded into and delivered by a variety of organizations types; in this study, two sites were non-profit social service agencies, two were county level public health departments, and one was an academic medical center. Sites were selected for participation by Hopelab and NFP, based on several factors that make up the diversity of the NFP program sites, such as geographic context (two urban, one suburban, one rural), staff size (three <6, two 12+), retention rate of clients (ranging from 65% to 85%), and how long they had been implementing the NFP model (ranging from 3 to 12 years). Other considerations for site selection included no involvement in current or anticipated research projects that would overlap with the pilot, and PHNs having access to work computers or tablets that could be used to access the dashboard.

Several mechanisms were used to ensure ethical oversight of the study. The study procedures were formally reviewed and approved by several IRBs, including the NFP’s internal committee for research, and an independent, external IRB, retained by Hopelab (Ethical and Independent Review Services, 2010). The approved study protocol was shared with site administrators early on during the planning phase for review and discussion about what additional institutional processes needed to take place. Subsequently, at the two sites that had institutional IRBs, the research team went through a process where all approved study procedures were reviewed by the institutional committees and granted an exemption based on ethical oversight elsewhere.

Participant Recruitment

Participants were 42 PHNs employed by the five NFP program sites where Goal Mama was piloted. All PHNs were implementing the NFP model; 83% (n = 35) in client-serving roles, and 17% (n = 7) in administrative and supervisory roles. A few nurses (10%) had been implementing the NFP models for more than a decade, 37% for 6–10 years, 39% for 1–5 years, and 14% had been implementing the model for less than a year at the beginning of the pilot.

Each site had a training session where PHNs were introduced to the Goal Mama platform and instructed on how to use both the app and dashboard in preparation for using the platform with their clients. At that training session, PHNs were invited to participate in the study for the duration of the 6-month pilot, and the study team described the study aims and driving research questions. The study team covered in detail what participation would consist of and answered questions that arose. They were reminded that their participation in study data collection activities was voluntary, that there would be no repercussions should they decline, and that they were free to withdraw from the study at any time.

A consent form was provided for participants to review, and sign if they wished to participate. At each focus group, and each time a survey was administered, participants were reminded that participation was voluntary, and they could choose to answer or not answer questions as they felt comfortable.

Data Collection

Participant observation

PHN participant observations were conducted at five training sessions, one at each of the participating sites. The trainings were conducted at the beginning of the study to introduce PHNs and their supervisors to the Goal Mama platform. Observational notes were taken using a standard note-taking form by a member of the research team at each site in order to assess PHNs’ initial reactions to the Goal Mama app and dashboard, what features resonated with them the most, and to record any questions they had about the platform.

Focus groups

A total of nine telephone-based focus groups (n range = 6–12) were conducted during the study in two distinct phases. In phase one, five initial focus groups were conducted (one at each site), approximately one month after training. These initial focus groups focused on understanding PHNs’ initial experiences downloading the app with clients and using the dashboard. In phase two, four monitoring focus groups were conducted with PHN supervisors (n = 7) from all five sites, once a month for four months. These monitoring focus groups focused on exploring barriers and facilitators to use in greater depth, as well as identifying changes to the app and dashboard design and features that could increase the acceptability, feasibility, and desirability of the Goal Mama platform for PHNs and their clients. We probed each focus group about what further education or knowledge was needed to effectively use and integrate the Goal Mama platform at their sites. See Supplemental Materials for sample focus group questions. The focus groups were audio-recorded, detailed notes were taken, and key sections of recordings were transcribed for analysis.

Surveys

PHNs (n = 42) completed a pre- and post-survey that assessed acceptability and feasibility of the Goal Mama platform at two time points. The pre-survey was conducted immediately after PHNs’ initial training but prior to actual platform use (T1). The post-survey was administered at the end of the study, following several months of platform use (T2). Feasibility and acceptability were measured using the Weiner et al Acceptability of Intervention Measure and Feasibility of Intervention Measure (Weiner et al., 2017). Scores for each item pair were averaged to create summary acceptability and feasibility scores (see Supplemental Materials). Cronbach’s alpha ranged from 0.85 to 0.91 for the four constructs (Cronbach, 1951). The surveys also included several open-ended questions. At T1, participants were asked for feedback on the training sessions, including what materials and group activities were most useful or could be improved. At T2, participants were asked to rate perceived helpfulness of the platform and provide open-ended product feedback about what was most useful and what they would change or improve.

Analysis

Our qualitative analytic process was iterative and spanned the course of data collection. We used a content analysis approach, given its established use in nursing research (Elo & Kyngäs, 2008) and the ability to apply the selected CFIR domains and constructs (see Table 1) to guide our a priori coding categories. Specifically, a directed content analysis expects researchers to use existing theories or prior research to come up with a preliminary coding schema, as well as develop new codes during analysis (Hsieh & Shannon, 2005). This approach supported our applying an existing theory (CFIR), as well as inductive processes, throughout the analytical phases of preparing, organizing, and reporting.

We began with preparing the data for analysis, reading through all data sources, creating memos of initial impressions, and taking note of emergent themes that could be added to our coding categories (Elo & Kyngäs, 2008; Sandelowski, 1995). We then moved into the organizing phase, conducting line-by-line thematic coding of all data sources (Bradley et al., 2007; Gibbs, 2007; Saldaña, 2015). All coding was completed using ATLAS.ti version 8 software (ATLAS.ti Scientific Software Development GmbH, 1993), and conducted by two members of the research team, with discrepancies brought to the larger team for discussion (Barbour, 2001; MacQueen et al., 1998). A priori codes were guided by CFIR domains and constructs, and as emergent themes surfaced during the coding process, these were also documented and applied as codes when salient. Our emergent coding was primarily low-inference (Anastas, 2004), following the literal meaning of respondents around key constructs of acceptability, feasibility, barriers, and facilitators.

We wrote analytic memos as a priori and emergent codes were applied and began to map key themes to the CFIR constructs (Bradley et al., 2007; Miles & Huberman, 1994; Saldaña, 2015). We utilized features within the Atlas.ti software (ATLAS.ti Scientific Software Development GmbH, 1993; Lewis, 2004) to identify the most salient thematic categories, reviewing and discussing code reports, frequency and co-occurrences of specific codes, and numerical and matrix displays of the coded data. We also continued to employ reflexive practices such as memo-writing and member checking (Ben-Ari & Enosh, 2011). As we reached saturation for salient themes from the qualitative data, we also triangulated results from the quantitative data.

For the quantitative survey data, we conducted multivariate regression analyses to examine whether PHNs’ perceptions (acceptability and feasibility scores) changed over time from T1 (immediately post-training and prior to beginning to use the Goal Mama platform) to T2 (study completion). Site fixed effects were included to ensure PHN survey and feedback data were compared within sites as well as across the sample. Analyses were run in the R statistical platform (R Core Team, 2000).

Results

Overall, participants were enthusiastic about the value and potential of Goal Mama from their first introduction to it. Over 6 months of use they identified critical areas for product and implementation improvement, but still viewed the platform favorably. These findings are summarized below, as key facilitators and barriers to pilot implementation of the Goal Mama platform.

Facilitators

Several key themes made the adoption of the Goal Mama platform feel highly feasible and beneficial to participants.

Technology helps to meet clients where they are and provides new insights for PHNs

Mobile technology is used extensively by NFP clients, and PHNs generally viewed Goal Mama as an important way for NFP to stay relevant to today’s young clients. PHNs shared that many clients have their phones in hand during visits and noted how integrated into clients’ daily lives mobile technology has become.

“[What’s most useful to moms is] the opportunity to set goals in their phone [that] is such a part of their daily lives” Nurse, T1 Survey

We also heard from participants that technology allowed for communication of topics or needs that may be uncomfortable to broach in person. As one nurse commented, directly after the training,

“[It gives clients] the ability to be transparent on a platform in a way that they might not be comfortable being in person.” Nurse, T1 Survey

There were various examples of this transparency about needs, ranging from a client using the app to request information on family planning and pregnancy prevention to a client disclosing via the app intimate partner violence as the topic that she wanted to address with her PHN at her next visit. The opportunity for communicating real-time needs on difficult topics to PHNs via the app helped PHNs’ understand unspoken client needs and prepare resources in advance.

Features provide value, skill-building, and social support

PHNs identified the goal-setting feature as impactful and an important place for their clients to identify and pursue meaningful goals.

“[Clients] have had the goals in their heads for a long time but they are working towards [them] and making progress faster.” Nurse, July 2018 Focus Group

“[The app] provides a way for moms to set goals and see them on screen. To write them out and think about them on a more regular basis.” Nurse, T2 Survey

This ability to see clients more easily set and pursue goals was attributed to the app’s step-by-step process to help clients clearly articulate goals, set milestones, brainstorm barriers that may arise and ways to address them, and monitor clients’ confidence as they work on each goal. PHNs reflected, throughout the course of the study, that having such a detailed process helped clients develop their goal-setting skills.

In addition, PHNs observed that the “Mom Community” feature provided a valuable sense of community and support from other first-time mothers. As one nurse noted,

“The community network [seems most useful for my clients.] I like the fact that the clients are able to receive support from each other.” Nurse, T2 Survey

In a program that is known for the client-PHN therapeutic relationship, the app provided an additional sense of belonging to a wider group of new mothers, above and beyond what a one-to-one relationship between the client and PHN could provide.

Supervisor modeling supports PHN platform use

PHN supervisors were key to Goal Mama adoption and sustained use. Supervisors reported using a variety of strategies, including one-on-one reflective supervision meetings, team meetings, and texting individual PHNs to keep Goal Mama top of mind as they began to use it with clients. Some supervisors tried to build the habit of using the platform by literally logging in together with PHNs.

“[The nurses] thought it was easy once they get on, and I have to work with them to get them on the dashboard.” Nurse Supervisor, September Focus Group

As the above supervisors noted, when they modeled consistent, engaged use of the platform, it reinforced use among PHNs and clients. However, the opposite was also true. During the same focus group, another supervisor from a different site commented,

“I can see a domino effect, since I haven’t been in it [the dashboard], then my nurses haven’t and their clients are not either. It is still new and not every client is in it and it’s still getting used to it and I have to integrate it into my days and weeks.” [Nurse Supervisor FG, September]

Supervisors with less engagement or whose engagement flagged at times, saw mirrored reductions in engagement from the PHNs they support. Participants clearly laid out that while supporting the introduction of the Goal Mama platform was one part of the equation, continuing to model its use was equally necessary.

Barriers

After spending several months utilizing the Goal Mama platform, participants identified key barriers to optimal use.

Designing for mobile needs versus desktop functionality directly informs use

The nurse dashboard for this pilot was designed based on the use case of a PHN having a day in the office where they prepared for visits and reviewed relevant information on a laptop or desktop computer. The pilot demonstrated that a more realistic use case was the dashboard being accessed in the field (including in the car prior to a visit, or during a visit itself) when a PHN did not always have a laptop or connectivity on their laptop.

“We are very mobile and don’t really go to a desk or computer every day.” Nurse, July Focus Group

“The only reliable and consistent technology that we have with us out in the field at this point in time is our iPhones.” Nurse, T2 Survey

In the T2 survey, which probed about desires changes to the platform, almost half of respondents reported a strong desire for a dashboard that they could access on their phone similar to the client’s app, and a belief that this would increase their use of the dashboard and be easier to incorporate into work routines. This need was particularly acute for field work and when visit preparation was frequently done outside of the office, such as in rural sites.

“I still wish we could use our smart phones to access the dashboard. Logging on my laptop using Google Chrome is an extra step that I am guilty of not taking some weeks. I would have accessed the nurse dashboard more if I could do so on my phone.” Nurse, T2 Survey

“Some of us don’t come in at the end, so I think of some nurses that are in rural settings so they don’t come into an office, and think about what it might be like for that experience and considerations of connectivity.” Nurse Supervisor, August Focus Group

Overall, we found that many PHNs across all NFP sites relied on their phones as the primary form of communication with their clients, as well as for preparing for and documenting their visits. A dashboard that wasn’t optimized for mobile use was not frequently used.

Limited tech and workflow integration strained PHN capacity

In focus groups and qualitative survey responses, more than a quarter of participants expressed significant barriers to integrating the Goal Mama app into their practice, and more than a third reported that more seamless integration would help increase and sustain use of the platform.

“You have to build it into your practice, so it becomes part of the routine. Once it’s NFP-wide that would make it easier. But it really needs to be built into the program.” Nurse, July Focus Group

Furthermore, the NFP program has several technological systems that PHNs are required to use, and which are already integrated into PHNs’ everyday workflows. Some sites had strict firewalls, lack of access to iCloud accounts, or organizational policies restricting the downloading of apps onto work phones. Solutions to these issues were found, but the effort required to address these barriers was substantial.

In this pilot phase of Goal Mama deployment, the Goal Mama platform added an additional technological system to the NFP program, and required PHNs to log in outside of their regular system, which resulted in their having one more thing to log into.

“It is really just the difficulty of too many things added for nurses to do, and not enough time to do them all.” Nurse, T2 Survey

Several PHNs further reported difficulty remembering to introduce, or check in about, the app due to the relatively small subset of their clients who were participating in the pilot and randomized to receive the app, and because the intervention itself was not yet integrated into their normal routines.

“I am a creature of habit and it is hard to remember to do something with a few clients only.” Nurse, T2 Survey

Nurse 1: “I think for me, it would have to start with a new client and have that start as something that we do every time; once we get in a routine with a client, it is harder to incorporate Goal Mama with an existing client.”

Nurse 2: “I would agree with that challenge, it is about building that into the relationship; an established client it is like ‘oh yeah, yeah, we do that at the end of the visit’; my client remembered to do that at the end, and I forgot about it.”

June Focus Group

Our data show that, given that the Goal Mama platform was not fully integrated into the workflow routine, additional steps such as (1) accessing a stand-alone digital platform outside their normal online systems or (2) keeping track of which few clients were part of the pilot, strained PHNs’ capacity and resulted in diminished use.

PHNs need additional support to feel confident using new technology

Participants reflected that it took time to develop comfort and familiarity with Goal Mama and said that building confidence would help increase adoption and sustained use. Lack of confidence and routine use of the platform prevented some PHNs from prioritizing bringing it up with clients, while others struggled to answer questions about it because they didn’t fully understand all of the product features themselves.

“[I needed to] take more time learning the app before clients started getting it. [I] need to feel more confident in what the app does.” Nurse, August Focus Group

“I am not confident in the app myself so I can’t really answer for others.” Nurse, July Focus Group

As the quotes above illustrate, several participants reflected that, to the client, the PHNs want to be the Goal Mama expert, but some PHNs need more support and experience using the product in order to serve that expert role.

Client reluctance and mixed perceptions about goal setting affected adoption

Participants reported that some clients were reluctant to use the app, either initially or over time, due to ways in which app use was perceived to be incompatible with clients’ current needs or life situations. Clients in crisis, those living in chaotic or uncertain circumstances, or those working to meet their basic human needs did not view “goal setting” as critical to their day-to-day realities.

“In my experience, it depends on how well they can access their executive functioning brain, a regular home visit might even be hard to get through.” Nurse, June Focus Group

“[Many of] our clients are just trying to survive, other things come first before they think about [something like] going back to school. . .” Nurse, July Focus Group

Although a case could be made that identifying and logging short-term goals and crisis-management steps via the app could be helpful first steps to meeting these pressing basic needs, PHNs were often inclined to solve for the crisis instead of utilizing the app. For example, if a client needed to locate a diaper bank to provide diapers, or apply for admission to an emergency shelter for safe housing, PHNs were unlikely to set those tasks up as discreet goals in the app, and instead, began troubleshooting real time to find the resources clients identified as their top priority.

App Usability and Feasibility

Analysis examining PHN feasibility and acceptability scores of the Goal Mama app and dashboard at two timepoints showed that immediately post-training (T1), most PHNs rated the app and dashboard as highly acceptable (all mean scores >4.6/5.0). Following approximately 6 months of app use with clients (T2), the acceptability and feasibility scores declined somewhat (all mean scores >3.4/5.0). All regressions testing the change from T1 to T2 were significant at the p < .01 level, indicating a reduction in scores across all four constructs from T1 to T2 (Figure 1).

Figure 1.

Acceptability and feasibility scores from nurses using the goal mama platform over two timepoints.

Note. T1: Nurses were surveyed after the nurse training at the Nurse Family Partnership sites has been completed; T2: Nurses were surveyed at the end of the study. Scores on all measures ranged from 0 to 5. The dashed line represents the maximum possible value of 5 and the error bars represent the standard deviation around the plotted means.

Discussion

As health systems and public health programs adopt digital processes, systems level buy-in and infrastructure are needed to foster sustained use and utility over time. The study reported here offers critical learnings from piloting a digital platform in an evidence-based public health program as a means of informing the implementation of digital interventions at scale. The potential for digital solutions is particularly relevant in COVID-19 aftermath, as health systems have rapidly sought technology-enabled solutions to reach patients during social distancing (Keesara et al., 2020). The lessons from this pre-COVID pilot are highly relevant and useful to sustaining the current wave of digital platforms rolling out across healthcare settings well beyond the current pandemic.

Participating PHNs in the 2018 pilot found the Goal Mama platform to be acceptable and feasible across 6 months of pilot use. The app leveraged mobile technology already in use by PHNs and their clients, facilitated communication and accountability between PHN/client partners, provided clients a safe space to set goals or broach sensitive topics on their own terms, and allowed both PHNs and clients to track progress over time. These findings are aligned with results from a community-based participatory research study exploring patient onboarding to a digital health platform, which found that mobile apps are best integrated into healthcare flow for low-income populations when it minimizes clinician workflow and burden to patients, and the role of health workers was fundamental to successful adoption (Liu et al., 2020).

Feasibility and acceptability scores were extremely high at the outset, suggesting a high likelihood of uptake and buy-in. Scores remained sufficiently high, though decreased, after 6 months, suggesting continued feasibility and acceptability with important opportunities for improving sustained use. These findings mirror results from other mobile app studies in the nursing context showing that usability, feasibility, and acceptability may be hindered when technical and other barriers to integration are present (Nordstoga et al., 2020; Redley et al., 2019). Qualitative data provided clear examples of what made the Goal Mama platform acceptable and feasible to use, while also providing insight into the specific barriers PHNs experienced, and helped the development team prioritize what needed to be addressed prior to wider dissemination.

Feedback and reflections from PHNs and PHN supervisors indicate that Goal Mama features generally aligned with client needs, and that the mobile app allowed clients to access these features using technology that was already a familiar and routine part of their lives. These facilitators appeared relatively straightforward. PHNs liked the product, believed in its value, appreciated its delivery system, and anticipated that their clients would benefit from it. Barriers, however, were more complex.

Technical challenges were a seemingly minor but fundamentally disruptive barrier. While not insurmountable (glitches can be, and were, fixed; connectivity issues can be worked around, etc.), these challenges are situated at the nexus of several other barriers, such as PHN capacity and workflow integration. PHNs are the first stop for clients’ technical assistance, which requires skill and comfort with the app, as well as the time to address client’s app-related issues. Making access to the app and dashboard as simple and intuitive as possible for both clients and PHNs is essential to ensure that the new tool doesn’t prohibitively strain workflow capacity by creating additional or duplicative tasks. Further, PHNs’ perception of reticence from clients who are struggling to meet their foundational or even survival needs, generates reticence in PHNs and represents a major barrier to onboarding particularly vulnerable clients.

Most barriers coalesced around what the CFIR framework calls trialability, meaning the ability of an intervention to be tested on a small scale (Damschroder et al., 2009). This showed up thematically as integration and capacity challenges, but, importantly, the lack of integration was due at least in part to the intentionally-limited scale of this pilot. Although unintentional, PHN capacity was strained by the additional effort required to work around it and add additional tasks to their role. Our data provided insight that new technology requires a reworking of routines and systems to ensure workflow integration, even within the context of a small pilot, in order to gather learnings that can be applied to roll out at scale. This insight is consistent with findings from previous studies demonstrating the unintended burden of piloting mobile apps for care providers (Ehrler et al., 2018; Liu et al., 2020; Redley et al., 2019).

Teams piloting new technology may benefit from explicitly considering innovative ways to build out mock integrated workflow strategies for initial testing in small subgroups where full backend integration is not yet feasible. This would help to address a common challenge in the health sector innovation space, and reduce the chance that high potential technology doesn’t result in an overly complex pilot. In hindsight, some of the strategies that may have benefited the Goal Mama pilot were related to directly creating tools to support simple utilization for PHNs. Examples of this include a minimal use experience that PHNs could use to build their capacity for using Goal Mama, even with few clients on it, or a crisis mode use case outlining ways to use specific features of Goal Mama to address clients with high-priority needs. The barriers to trialability identified through this pilot provide useful insight for future projects to consider when designing pilot infrastructure.

Furthermore, the study highlights the importance of designing technology with safety and privacy in mind (BinDhim & Trevena, 2015; Huckvale et al., 2015; Lustgarten et al., 2020). The disclosure of potential sensitive information (e.g. interpersonal violence) via mobile app underscores the critical need for developers to create data protection policies and infrastructure that uphold the rigorous protection of health information. Previous research on intimate partner violence has found mobile apps designed with safety and privacy in mind to be an acceptable and feasible supplement to in-person health care visits, providing a rapid way to disclose and engage in safety planning (Alvarez et al., 2018; Tarzia et al., 2017). Developers and researchers alike must continue to simultaneously consider how a mobile app can facilitate enhanced communication between patients and providers, while also ensuring said communication remains confidential to the appropriate security standards (e.g. HIPAA).

Limitations

Our study had some limitations. The number of sites studied here was low relative to the total number of NFP sites, but they were chosen to be diverse with respect to size, geography, and socio-demographic characteristics of the clients served. Additional barriers or facilitators may emerge as the intervention is more widely disseminated that were not captured in this pilot due to the limited number of sites. Furthermore, the purposive sampling approach to site selection based on certain criteria, including interest in piloting new technology, introduces the possibility of selection bias and may limit the broader relevance of these findings.

Implications for Research and Practice

In maternal health, use of mobile apps by patients is ubiquitous and gaining traction among providers and health systems. Mobile technology may be particularly valuable for programs that are serving an increasingly digital native population. However, acceptability, feasibility, and usability remain key challenges from a health systems perspective. A more nuanced understanding of how barriers play out and what possible strategies are to address them is needed.

This study provides new evidence on how to address barriers and leverage facilitators to support implementation of technology-based solutions within national home-visiting programs.

Strategic integration, including minimal disruption of clinical flow and enhancing communication between providers and patients can reduce barriers and promote usability for providers and clients. The findings were used to provide recommendations to NFP regarding their national roll out of the Goal Mama platform to the approximately 270 sites across the United States. Many of these recommendations were acted upon immediately, including the design and development of a mobile version of the nurse dashboard and training materials that emphasize how goal setting and task tracking can be essential during times of crisis.

The lessons learned in this study are relevant and valuable to the various systems and stakeholders who are invested in ways to leverage digital technologies to augment in-person care, including front-line program staff, service providers, administrators, and evaluators.

Conclusion

Overall, PHNs found Goal Mama to be both an acceptable and feasible addition to their work supporting clients as they navigate pregnancy and parenting, and highlighted the key barriers that need to be addressed for sustainability beyond a 6-month pilot. It is critical to monitor how scaling up the use of the platform can prioritize the technological and programmatic integration needed to lower key barriers and further facilitate implementation.

The high feasibility and acceptability ratings of the Goal Mama platform by participants, and willingness to incorporate it into their practice, shows the potential for the platform to improve efficiency and coordination of visits, as well as client evaluations of the program over the long term. Feedback from participants on how to improve the integration of Goal Mama into practice suggests an important role for administrators and PHN supervisors in providing ongoing technical and behavioral guidance and support. Scaling at the national level will require a thoughtful technology integration and PHN education plan, and implementation efforts should continue to be evaluated.