The development of a theory-based eHealth app prototype to promote oral health during prenatal care visits

Nurse practitioners and midwives reported that an eHealth app prototype assisted in providing them with the information, motivation, and skills needed to implement the oral health guidelines into routine prenatal care visits.

Abstract

Poor maternal oral health during pregnancy is associated with adverse maternal and child outcomes, including preterm birth and early childhood caries. Subsequently, professional associations have developed prenatal oral health guidelines, but significant gaps exist in implementing guidelines into clinical practice. The purpose of this study was to develop and test the usability of an innovative, theory-driven, eHealth application (“app”) to facilitate prenatal providers’ (nurse practitioners and midwives) implementation of oral health promotion during prenatal care visits. App development was guided by previous research, an integrated conceptual framework, Scientific Advisory Board input, and consumer-engaged iterative processes utilizing mixed-methods (observations, surveys, in-depth interviews) among providers (n = 4) during 10 unique prenatal care visits at a federally qualified health care center. Triangulation of quantitative and qualitative data analysis produced descriptive frequencies and salient themes. Concepts and principles from the following theoretical frameworks informed intervention development and testing: Consolidated Framework for Implementation Research; Information–Motivation–Behavioral Skills Model; Health Literacy; and Brief Motivational Interviewing. Overall, providers reported the app was effective at providing the information, motivation, and behavioral skills needed to integrate oral health promotion (e.g., easy to use; provided cues to action via scripts and tailored education; and documented findings into the patient’s record). Although providers reported high usability, time constraints and detailed patient counseling scripts were identified areas for improvement. Findings suggest that the eHealth app could serve as an innovative mechanism to assist providers in implementing the prenatal oral health guidelines into practice. Future research is needed to continue app development efforts and to determine efficacy and effectiveness in practice settings.

Implications

Practice: An evidence- and theory-based eHealth app could serve as an innovative mechanism to assist prenatal providers in translating the prenatal oral health guidelines into clinical practice.

Policy: Development and implementation of an app must consider organizational characteristics, including routine workflow, cost of implementation, and context-specific concerns.

Research: Future research should conduct rigorous randomized control trials among larger samples and different prenatal care settings to assess the impact of an eHealth app across diverse clinical contexts.

INTRODUCTION

In the USA, oral disease is a silent epidemic [1]. Specifically, among pregnant women, approximately 40% experience periodontal disease [2]. During pregnancy, hormonal changes increase a woman’s vulnerabilities to bleeding and gingival inflammation, both increasing the risk of gingivitis and periodontitis [3,4]. Poor oral health conditions, such as periodontal disease, have been associated with a range of adverse pregnancy and birth outcomes (APBOs), including pre-eclampsia, gestational diabetes, preterm birth, and low birth weight [5,6]. Additionally, maternal oral health behaviors can impact a child’s risk of early childhood caries [7–12].

Given the significance of oral health during pregnancy, the American College of Obstetricians and Gynecologists (ACOG) and the American Dental Association (ADA) have coendorsed national guidelines: Oral Health Care During Pregnancy: A National Consensus Statement [13]. This guidance specifies key practice behaviors for both prenatal and oral health providers, such as: assess patients’ oral health status; advise patients on oral health hygiene habits and behaviors; and refer patients to prenatal/oral health providers and collaborate on ongoing patient care [13]. Although approximately 83% of U.S. women receive prenatal care in the first trimester, only half (48%) receive oral health care during their pregnancy [14]. Subsequently, various barriers exist in implementing the prenatal oral health guidelines into comprehensive and quality care, including few prenatal providers believing they have the time or training to follow the recommended behaviors [15–17].

eHealth technologies can significantly improve healthcare processes, outcomes, and quality of care, and could serve as an innovative solution to address barriers and persistent gaps in providers’ practice behaviors [18–22]. eHealth is commonly used in the clinical setting for communication and electronic record keeping [23–26] and to improve clinical outcomes through applications (apps) focused on education, promotion, and decision-making processes. eHealth technologies can improve guideline adherence [27–29], patient–provider communication [20], health information comprehension [21, 30, 31], and patient activation [22,24,30]. eHealth apps, such as those installed on tablet computers, can be interactive, incorporate multiple learning strategies, and have a greater potential to educate users compared to other methods of health promotion [32].

Although the significance of prenatal oral health and the utility of eHealth technologies in improving quality of care during clinical interactions has been documented, less attention has been devoted to designing and assuring that theory- and evidence-based concepts and principles are translated into practical interventions that aim to improve guideline implementation. In addition, it is critical to engage consumers and to conduct formative research necessary to understand the context and system-level implementation factors that may influence the adoption and use of health interventions. Thus, the purpose of the current study was to develop and test the usability of a theory-driven, provider-centered eHealth application (“app”) to facilitate prenatal providers’ (nurse practitioners (NP) and midwives; “providers”) ability to assess, advise, and refer pregnant patients on oral health issues. Capitalizing on the advantage of eHealth technology, findings from this study will provide important insight into the app’s capabilities in translating guidelines into clinical practice and improving quality of care.

METHODS

This study utilized formative research, evidence- and theory-based constructs and principles, input from a Scientific Advisory Board, and engagement with end users (providers) to develop and pretest an eHealth app prototype. This eHealth app prototype was a web-based program that was delivered on an iPad that providers used during a portion of the prenatal care visit with their patients. The Scientific Advisory Board included researchers and clinicians with expertise in prenatal care, oral health, patient–provider, eHealth communication, and health literacy. The research for this study was conducted at a Federally Qualified Health Care Center (FQHC) that delivers comprehensive health and social services (e.g., prenatal, dental) across several clinic sites to underserved communities in the southeast region of the USA. Institutional Review Board approval was obtained prior to the study.

Conceptual framework of eHealth application development

The main theoretical framework guiding the study was the Consolidated Framework for Implementation Research; concepts and principles from the following theoretical frameworks also guided intervention development and testing: Information–Motivation–Behavioral Skills Model (IMB); Health Literacy; and Brief Motivational Interviewing (BMI). See Fig. 1 for the integration of these theoretical frameworks which contributed to the study’s conceptual framework.

Fig 1.

| Conceptual framework guiding development of eHealth app to promote prenatal oral health.

Consolidated framework for implementation research

Consolidated Framework for Implementation Research (CFIR) is a meta-theoretical framework that draws from existing evidence-based theoretical concepts within the field of implementation science [33]. Specifically, it consists of a menu of 39 constructs across five domains: Intervention Characteristics; Inner Setting; Outer Setting; Process; and Characteristics of the Individuals Involved [33]. A formative phase of this study assessed key implementation constructs to inform the development of this intervention via in-depth interviews and focus groups with clinical administrators, IT personnel, prenatal and oral health providers; clinical observations; and assessment of current IT infrastructure. In addition to informing end users’ (prenatal providers) preferences with regards to intervention characteristics, these data provided valuable insight into the multilevel contextual factors of the clinical setting to which the intervention would be implemented [34].

Guided by the study’s conceptual framework and the formative phase of this study, the team created a mockup of the eHealth app to be used in the usability stage of the project using concepts and principles from IMB, health literacy, and BMI [34]. The IMB model posits that information, motivation, and behavioral skills are key determinants of preventive health behavior [35]. These constructs guided the development of the intervention components and functions that would subsequently alter providers’ psychosocial determinants and their practice behaviors (i.e., assess, advise, and refer patients on oral health issues during prenatal care visits).

Health literacy is defined as the “degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions” [36]. Health literacy is recognized as a key determinant of health, and similar to the IMB Model, one’s knowledge, motivation, and competence can impact skills needed to engage in positive health behaviors [37]. Moreover, health literacy is influenced by individual and systemic factors, including communication skills and demands of the situation and context [36]. Health literacy principles also guided the intervention development, such as facilitating providers’ ability to access and understand basic oral health information that is clear and actionable. In addition, the intervention facilitated providers’ ability to appraise information received from the patient’s health history and apply information through patient-centered communication techniques [37].

BMI is a communication approach that emphasizes patient-driven conversation and empathic listening to increase patient’s personal motivations for change [38]. The intervention used BMI concepts and principles to facilitate patient-centered communication (one of the health literacy domains), such as understanding the patient’s oral health history, needs, and concerns, and collaborating on behavioral action plans based on the level of importance and confidence that the patient has with regards to oral health [38].

The app featured four main sections: (a) assess—provider queries patient on oral health history and visually inspects the patient’s mouth; (b) advise—provider educates the patient on the importance of good oral health during pregnancy and advises on positive oral health behaviors the patient can do to promote oral health during pregnancy; (c) refer—provider refers the patient to an oral health provider; and (d) summary—production of a provider summary sheet to be included in the patient’s electronic health record and a printed patient summary sheet to take home (Fig. 2).

Fig 2.

| Example screen shots of eHealth app prototype.

The app components and content were designed to give providers the necessary information, motivation, and behavioral skills needed to implement the prenatal oral health guidelines into routine prenatal care visits. For instance, to motivate the provider to engage in the recommended practice behaviors (i.e., assess, advise, refer), the provider-centered app was designed to (a) address the previously identified barriers to guideline implementation (e.g., lack of oral health knowledge and skills; lack of time; not part of routine practice) and (b) capitalize on the advantages of using technology during patient interactions (e.g., quick; easy to use; facilitates patient–provider communication). The app also provided tailored scripts with the option of expanding scripts for more detailed information depending on provider’s knowledge level and extent of information needed to answer potential patient questions, as well as embedded features such as pop-up pictures during the assessment to assist providers in conducting the visual oral health screen to identify potential problems and issues that would be later referred to the oral health provider.

Sampling and recruitment

A convenience sample of providers (NPs and midwives) and pregnant patients were recruited through the assistance of the clinical partner. Research staff members briefly oriented each provider to the eHealth app and obtained their informed consent to participate prior to data collection. Additionally, research staff members worked with the clinic’s front desk staff to identify English-speaking pregnant patients. Potential patients were approached, informed about the study, and if they were eligible and interested in participating, they provided signed informed consent. No other information was collected from patients, as they were not the primary targets for this phase of the research. Patients received a $30 gift card for their time (allowing researchers to observe a portion of the prenatal care visit). Providers did not receive individual incentives as the clinic received an honorarium as part of the larger parent study to compensate for organization’s participation and contributions of time, space, personnel, and other resources. Four providers matched to 10 pregnant patients were included in this study.

Data collection methods

Observations

Observations were conducted during a portion of the prenatal care visit where the provider used the app with their patients. The app was a web-based program that was delivered on an iPad and was integrated into the clinic visit. Providers were given an overview of the app prior to using it with their patients. The provider logged on by entering their name and the patient’s gestational age. The provider then followed the app’s prompts and scripts through the sections (assess, advise, and refer) while interacting with the patient. A “talk aloud” approach [39] was encouraged where providers verbally explained their processes and thoughts while navigating through the eHealth app. Research staff audio recorded and took detailed notes during the observation. Examples of the scripts from the app are presented in Fig. 2.

Survey

Following the observation, each provider completed a brief survey. The survey consisted of questions on a five-point Likert scale assessing the usefulness of app and the app’s ability to provide them with the information, motivation, and the behavioral skills needed to engage in the prenatal oral health practice behaviors. The system usability scale (SUS) was also administered as a global measure of satisfaction with the app. The SUS is a widely used instrument in the field of usability research with high reliability and validity and recommended by the U.S. Department of Health and Human Services to measure usability with small sample sizes [36]. This survey includes 10 items with responses based on a Likert scale.

Follow-up interviews

Following the surveys, each provider participated in an individual 30 min interview. Providers were asked questions about what they liked and disliked about the app, their understanding of the app’s functions, and their thoughts on the app’s ability to help them implement the prenatal oral health guidelines and communicate to patients during prenatal care visits.

Data analysis

Researchers summarized data and identified salient themes from the patient–provider observational sessions through listening to the audio recordings and reviewing field notes. Data from the follow-up survey and the usability items were entered into Excel and frequencies were tabulated. Audio recordings of the interviews were transcribed verbatim. A codebook was developed based on a priori themes guided by the conceptual framework as well as the aims of this phase of the larger parent study (assess the functioning and usability of the app prototype during a real prenatal care visit).

RESULTS

Demographics

All providers were female and had a median age of 49 (37–62 years; SD ± 12.12). Three providers were white and one African American. Two were certified nurse midwives, one an advanced NP, and one registered as both. On average, providers had been employed for 14.75 years (3–22 years; SD ± 8.54) and had been employed by the FQHC for 2.5 years (0–4 years; SD ± 1.73).

Observations

Overall, all providers were able to navigate the app easily and confidently after orienting themselves while working with their first patient. During the visual assessment, Provider 4 interacted with the patients the most, asking the patient questions about their health and informing the patient what they were seeing to corroborate the patient’s oral history. Provider 3 used the pictures in the app the most, using them as a reference point when they were not sure if they had spotted a potential infection or problem area.

During advisement, multiple providers correctly asked the patient about their beliefs regarding oral health hygiene behaviors but sometimes did not follow the script and used leading language such as “You do that right?” Instead of asking about their confidence related to oral health hygiene skills, some providers ran through them as commands versus recommendations. One patient with Provider 2 had particularly low oral health knowledge and the provider was able to use the expanded script feature of the app to discuss why there is an increased vulnerability for poor oral health during pregnancy. Provider 3 used the open-text box notes feature of the app for all patients, even if it was only to record that there were no concerns. Provider 3 also advised patients on oral health and accurately noted information regarding their beliefs and confidence in oral hygiene behaviors. Provider 4 was the only provider to not use the tailored scripts, choosing to summarize and reword the scripts in words they felt most comfortable.

All providers were able to refer their patients to oral health services. Although one provider used the notes section often, others did not use embedded documentation tools beyond ticking the basic boxes throughout the program, choosing not to record any patient concerns, including gaps and concerns regarding the lack of patient knowledge and health behaviors and potential physical symptomology observed. This limited the usefulness of the provider and patient summaries produced at the end of app experience.

Surveys

Sections/functions

When surveyed about the app’s sections and functions, providers considered most of the sections to be “useful” or “very useful” (see Table 1). The provider and patient summary pages were the only sections where one provider responded that they were “not very useful.” Two providers indicated they did not use the resources section of the app, but two others indicated that they found the section “very useful.” Most found the suggested scripts to be helpful in introducing the topics to patients. Half the providers did not use the notes feature, while the other half found them very useful.

Table 1 .

| Usefulness of app sections and functions (n = 4)

	Not very useful	Neutral	Useful	Very useful	NA/not used
Program overview			1 (25%)	2 (50%)	1 (25%)
Assess section			1 (25%)	3 (75%)
Advise section			1 (50%)	2 (50%)
Referral section		1 (25%)		3 (75%)
Provider summary	1 (25%)			3 (75%)
Patient summary	1 (25%)			2 (50%)	1 (25%)
Resources				2 (50%)	2 (50%)
Pop-up pictures (“image” icon)				3 (75%)	1 (25%)
Suggested scripts to introduce main topics			1 (25%)	3 (75%)
Tailored scripts based on patients’ responses		2 (50%)		2 (50%)
Print function			1 (25%)	2 (50%)	1 (25%)
Email function				2 (50%)	2 (50%)
Response buttons (yes/no)				4 (100%)
Open-ended response fields (text entry field)			1 (25%)	2 (50%)	1 (25%)

Providing information

Overall, providers “agreed” (25%) or “strongly agreed” (75%) that the app was effective at providing information regarding the importance of oral health promotion during pregnancy (see Table 2).

Table 2 .

| App’s ability to provide the information, motivation, and behavioral skills for oral health promotion during prenatal visits (n = 4)

	Strongly disagree	Disagree	Neither agree or disagree	Agree	Strongly agree
The app was effective a providing information about…
…the importance of oral health for the overall health of my pregnant patients.				1 (25%)	3 (75%)
… the association between poor oral health and adverse birth outcomes such as low birth weight and preterm birth.				1 (25%)	3 (75%)
…oral health symptoms that may be present during pregnancy.				1 (25%)	3 (75%)
…the importance and safety of seeing a dentist during pregnancy.				1 (25%)	3 (75%)
… how bacteria can transfer from mother to baby, increasing risk for early childhood caries.				1 (25%)	3 (75%)
… the oral health during pregnancy practice guidelines from professional organizations.				1 (25%)	3 (75%)
After using the app, please indicate the degree of your agreement with each of the following items:
This app increased my motivation to address oral health with my pregnant patients.			1 (25%)	1 (25%)	2 (50%)
This app helped me to communicate about oral health with my patient.				2 (50%)	2 (50%)
This app helped me to foster shared decision-making regarding my patient’s oral health behaviors/care.			1 (25%)	1 (25%)	2 (50%)
This app would be useful for addressing oral health with my pregnant patients.			1 (25%)	1 (25%)	2 (50%)
This app would fit well into my daily practice routine.		1 (25%)	3 (75%)
I am likely to use this app in my practice.	1 (25%)		2 (50%)	1 (25%)
After using the app, I am confident that I can...
…take an oral health history from my pregnant patients by asking them questions.				2 (50%)	2 (50%)
…check for potential problems or signs by looking in the mouth.		1 (25%)		2 (50%)	1 (25%)
…document my assessment findings in the app.		1 (25%)		2 (50%)	1 (25%)
…advise my patient about the importance of oral health.				3 (75%)	1 (25%)
…advise my patient on good oral hygiene and other health behaviors.				3 (75%)	1 (25%)
…refer my pregnant patient to an oral health provider.			1 (25%)	1 (25%)	2 (50%)

Motivation and communication

Providers generally “agreed” or “strongly agreed” that the app increased their motivation and helped them communicate about oral health with their patients. However, when asked if the app was suitable for their daily practice, most participants were neutral and one provider disagreed. Similarly, providers were asked if they were likely to use the app in their daily practice and the results were mixed.

Confidence

Providers “agreed” or “strongly agreed” that they were confident in their ability to assess, advise, and refer their patients on prenatal oral health issues after using the app. However, one provider expressed not feeling confident in their ability to conduct visual assessments in the mouth nor in their ability to document their findings.

Usability

The final survey component was the 10-item SUS. To interpret the individual SUS survey, the scale is converted to a score ranging from 40 to 100 points. Research based on previous studies has found that the average SUS score is 68 and scores above or below this indicate high or low usability, respectively [36]. On average, the eHealth app scored 80.6 (±11.25), indicating a generally high usability. Providers were in agreement in finding the app to be easy to use, consistent, and expressing confidence in their ability to use the app. However, only one agreed they would use the app frequently (Table 3).

Table 3 .

| System usability scale

Question	Average scorea (M ± SD)
1.I think I would like to use this app frequently	1.50 ± 1.29
2.I found the app unnecessarily complexb	2.75 ± 0.96
3.I thought the app was easy to use	3.50 ± 0.58
4.I think that I would need the support of a technical person to be able to use this appb	3.75 ± 0.50
5.I found the various functions in this app were well integrated	3.50 ± 0.58
6.I thought there was too much inconsistency in this appb	3.50 ± 0.58
7.I would imagine that most people would learn to use this app very quickly	3.50 ± 0.58
8.I found the app very cumbersome to useb	3.50 ± 0.58
9.I felt very confident using this app	3.25 ± 0.96
10.I needed to learn a lot of things before I could start using this appb	3.50 ± 0.58

^aEach item scored on five-point Likert scale, 1 = lowest and 5 = highest.

^bItems Q2, Q4, Q6, Q8, and Q10 are worded reversely and were recoded for consistency using the equation: 5-original score.

Follow-up interviews

General merits

All the providers found that the information in the app was valuable and was presented clearly. Providers discussed how they could easily navigate through the app and liked the design, including app navigation, font, size, and color. Providers also shared their appreciation for several app features, including program introduction that included a summary of the national guidelines, step-by-step assessment questions, and the ability to use both response buttons and open-ended fields to record answers. In addition, providers found the picture pop-ups to be a useful aid in conducting the oral assessment as it provided them with visual references to differentiate between normal and potential abnormal physical manifestations (Table 4).

Table 4 .

| Provider interview themes

Themes	Supporting quotes
Most liked features	“The pictures were great. That’s what I wanted, ‘cause I’m not a dentist. I don’t know what I’m supposed to be looking at.” “I liked the first screen asked the questions, have you had this, this, yes or no checkbox for the assessment.”
Least liked features	“To me, it seemed-I mean it had good information, but it seemed wordy. Like I was just reading from the screen. I guess I would have to try and put it into my own words or something.”
Unused features	“I didn’t realize there was a picture. No one said anything about a picture icon.” “Just if there’s a tutorial how to use it or something…I kind of [just went] went click, click, click.”
Usefulness of the provider and patient summary	Provider Summary: “I think it’s kind of redundant.” Patient Summary: “It gave the patients something that they can take home and look at later and recap what we talked about.”
Usefulness of brief motivational interviewing	“I think all of the things that you asked them were very reasonable to do…those are easily accomplishable goals…give them a visual thing” “I’m not sure you’re gonna get people motivated to spend money out of their own pocket to go to the dentist…Because you got to hook them. And as soon as they walk out that door, they’re not gonna think about the dentist anymore”
Patient–provider communication	“It reminds me to hit the hot spots…” “I think it’s a bit lengthy and I think after about the third question that I kind of lost them, a lot of them. You’re just kind of wandering off in space”
Merits of the app	“It prompts us to be a little more in detail, especially the pictures. Those were great.” “There’s just the right amount of information. I wouldn’t give any more or any less.”
Demerits of the app	“I felt like an idiot when I was going over those ‘That’s great’ [in the Advise section]. That was a little childish.” “Those appointments take forever. So, it’s a little annoying to be doing another piece of a puzzle, which is appropriate, but again, it’s just time consuming.”
Suggestions for improvement	“I think you should have some follow up with pictures to give the patients …’cause they’re looking in the mirror and they don’t know what’s normal and what’s abnormal.” “Maybe [do it] on intake, that way you’re not quite as crunched for time, you can go over it.”

Provider knowledge

Providers believed having the app would be a more accessible and reliable source of information during clinical interactions compared to relying on “Google” to look up information. Another provider mentioned that they would like the app to provide more information on oral cancer, indicating a need and the potential for the app to serve as a reference guide for a range of oral health issues.

Provider motivation

While some providers felt that they would have included oral health discussions without the app, others thought that having the app helped them delve into oral health in greater details than they normally would. One provider expressed that they would be motivated to use the app as it could influence their patients’ perception of the importance of oral health. This provider believed it would make patients more aware of oral health and its significance during pregnancy as they are witnessing the provider taking this topic seriously, given their use of the app and explicitly integrating this topic into their prenatal care visit.

Patient–provider communication

Some providers mentioned that since they would not have otherwise asked certain oral health questions to the patients, the app acted as both a cue to action and a patient–provider communication tool. Providers also reported that the tailored responses provided in the app were very useful while communicating with each unique patient.

Time intensive

Providers expressed concerns about a few aspects of the app that made it time consuming. Some providers believed that they might have integrated some oral health conversations without the app and in less time, as the app prompted them to share comprehensive scripts, which they felt they could not abbreviate given their participation in this study.

Tailored script/information prompts

Related to the perceived time-intensive nature of the scripts mentioned above, one provider felt that the app had a repetitive and “corny” structure, requiring them to give the same information to the patient multiple times. Another provider mentioned they felt the need to skip a few questions because they thought they were redundant and did not contribute anything to their patient interaction. This redundancy was mostly noted in the advise section, where the app prompts the provider to assess the “importance” and then “confidence” that the patient holds regarding various criterion-specific practice behaviors as guided by BMI principles. These providers discussed that they would rather use their own abbreviated words and that some of these advisements required prior education before they asked the patient about their importance and confidence to do a specific behavior (e.g., rinsing their mouth with baking soda instead of brushing their teeth with toothpaste after vomiting to protect their teeth).

Some providers felt that the BMI technique was a good way to open dialogue about potential barriers to oral health behaviors. For example, if a patient responds negatively when asked if she is confident about a particular oral health behavior, the provider could guide them in taking positive steps toward the targeted behavior. Nonetheless, other providers had doubts about the usefulness of this technique. Multiple providers suggested that it would be better to provide dental referrals instead of relying on the patient to change oral health behaviors based on the discussion during prenatal care visits and to only provide passive oral health education during this time.

Patient–provider summary

Most of the providers thought that the provider summary was not of any use, believing that they as providers would not revisit the notes. All of the providers agreed that the patient summary would be beneficial to the patient, as it would reinforce what was discussed.

Unused features

Two providers reported not using the embedded pop-up pictures, as they did not recognize the symbol to click on to show these pictures. However, after being oriented to this feature, they said it would have been very useful if they had known it was available. While most providers liked the feature of asking and reporting open-ended questions, one provider reported not using these field options, as there seemed to be “nothing to report.”

Suggestions for improvement

In addressing the difficulty that providers experienced with recognizing the picture icon, one of the providers suggested including an “examples” caption below the picture buttons to encourage providers to click on the link. Another suggestion was to develop the app in multiple languages as at this particular study site there were many Spanish-speaking patients who would benefit from the information in their native language. Some providers had general suggestions, including putting the pictures used in the app on the clinic walls to give patients a better idea of what to look for during oral hygiene care at home. They also felt it would be beneficial to give patients a copy of the pictures to take home alongside their patient summary. The most common suggestion for improvement was to lessen the time burden on the providers. Providers recommended either shortening the time the app takes from approximately 6 to 2 min (or less) or remove the responsibility from the provider by having other health care staff (e.g., medical assistants, nurses) conduct the assessment and advisement sections.

DISCUSSION

This study provided rich information and preliminary evidence that an eHealth app can assist providers in assessing, advising, and referring pregnant patients on oral health issues. Several strengths and potential areas for improvement were also revealed and will guide future research and development efforts. Findings indicated a high acceptability of the technology among providers in this study (NPs and midwives), indicating that the app is an appropriate modality for future iterations of this intervention. Previous research has examined health care providers’ viewpoint on eHealth apps, mainly regarding medical records systems and Clinical Decision Support Systems (CDSS) and corroborate that providers are favorable to eHealth apps and acknowledge them as useful for collecting patient health information [40–44], making health risk assessments [45], and improving communication [20]. eHealth implementation studies have identified several factors for successful implementation, including complexity of program, cost, compatibility with the organization’s workflow, infrastructure for training and maintenance, and engagement of stakeholders [46]. Along with further development and testing, future research should explore the cost and provider training needs associated with adoption and use of this app into routine clinical care.

The SUS, a validated tool, was used in this study to examine the app’s global usability for routine practice. Specifically, usability in eHealth helps to establish if the app is efficient, effective, and compatible into routine workflow, and the overall user’s satisfaction [47]. If an eHealth app is expected to be adopted into routine practice, it must be proven to be usable by the targeted population and is best measured through a reliable evaluation tool [36]. With a score of 80.6, the app in this study tested with high usability, 12.6 points above the average SUS score of 68 [36]. High usability of eHealth applications, such as what was found in this study, indicates that the app is easy to navigate for new users and is easy to relearn if practice guidelines change or if the app is modified [47]. This is critical for future app iterations and its ability to be adapted to different clinical settings and groups.

This study recognizes that providers need to be the recipients of health literacy efforts before they can be agents and improve oral health literacy among their patients. The majority of providers in this study agreed that the app provided them with the information, motivation, and behavioral skills needed to integrate oral health promotion during prenatal visits. These providers believed the app provided them with key oral health information at differing levels of details via the option to expand text if they were “stuck” in sharing more detailed information about that topic with patients. Previous research has found that most prenatal providers lack oral health knowledge, a critical reason for not providing oral health promotion to their pregnant patients [15,17,48–50]. As guided by the IMB Model, improving knowledge is a fundamental determinant and can impact providers’ motivation and confidence to engage in practice behaviors. While not all providers in this study used the scripts verbatim, all used the content and were able to tailor it to their preferences. Previous research has also shown that tailored scripts can help providers feel more comfortable, and therefore more confident, in talking about prenatal oral health. Such cueing can strengthen providers’ ability to provide clear and consistent messaging when discussing specific health topics [51,52].

Low knowledge often contributes to low self-efficacy, and providers have previously reported that they do not discuss oral health during prenatal care because they lack the knowledge and skills to perform oral health assessments or advise their patients [48,50]. While the providers in this study initially felt uncertain about their ability to visually assess patients’ oral health, the visuals aids embedded in the app served as an important tool for increasing self-efficacy and improving their ability to recognize potential warning signs of poor health. The use of visual aids has been an established tool to improve knowledge and risk communication with patients [36,53]. With many providers unsure of their role and ability to address oral health, the use of visuals provides a clearer idea of what may be normal versus a sign of a potential problem, making it easier to conduct assessments with confidence.

Findings also revealed that providers believed the app was useful in improving their patient–provider communication, including influencing their motivation and ability to collect and share oral health information. Previous eHealth apps have been found to improve patient–provider communication by providing key discussion points that are important to bring up with their patients [46,54–57]. Effective patient–provider communication is critical in all health care settings but is especially important in the prenatal setting when patients are motivated to make behavior changes and report their providers as the most influential and trusted source of health information [58–61]. However, future research should also consider how the app could be modified to incorporate provider feedback received during this usability testing, especially regarding the time burden and tailored script concerns. For instance, future research and development efforts should attempt to revise scripts and use concise prompting to decrease the amount of text on the screen, while upholding best practices in patient–provider communication principles. In addition, future efforts could explore developing an accompanying provider training module that could eliminate some of the current information that is present in the app. In this manner, the app could focus on prompting and providing brief messages and cues to facilitate patient–provider interaction, whereas the training module can provide expanded background information so that the providers are appropriately educated and trained on this topic prior to their patient interaction.

Findings from this study should be considered given noted limitations. First, this study was conducted at one FQHC and may not be generalizable to all prenatal care settings. Second, this health center has dental services within its organization, and thus providers may be more familiar with referring patients to a dentist. Similarly, providers may have gone through the app more quickly, making assumptions on the information and care patients have or will receive, given this clinic’s standard practice of trying to link all pregnant patients to dental services. Third, usability testing was conducted among a very small sample of prenatal providers (n = 4), which included contracted certified nurse midwives and an advanced NP. Although this study permitted preliminary feedback during real patient–provider clinical encounters, findings are not generalizable, particularly to other types of prenatal providers, such as those with medical degrees (e.g., MD, DO). However, these contract providers reported working in other prenatal care settings and thus could have considered these other clinical environments and experiences when responding to questions on the survey and during the follow-up interview.

Nonetheless, these limitations are countered by multiple strengths of the study. This study was novel through its application of an integrated conceptual framework, which consisted of constructs and principles from several theoretical frameworks that guided the app development and feasibility testing. eHealth apps aimed at behavior change are often critically lacking a foundation in theory, detrimental to the app’s ability to make effective change [62]. In addition, the app’s development and testing featured a strong multidisciplinary team, including researchers and consultants from public health, medicine, dentistry, health communication, health literacy, and technology. Thus, this study bridges research and practice by translating evidence and theory into a practical intervention to facilitate oral health promotion. Moreover, this translational science study incorporates the consumers throughout the research and development process, while acknowledging the implementation context. Future research building on this intervention should consider using the app in different settings and populations to validate the study results and improve the app’s utility. Moreover, future research should conduct rigorous randomized control trials among larger samples and different prenatal care settings to assess its impact across diverse clinical contexts.

This study also demonstrates the importance of explicitly using evidence and behavioral theory when designing eHealth interventions. Translating theoretical constructs, methods, and principles into the user interface, content components, and graphic elements ensures the app stays true to its “theory of logic” and will form the basis of future evaluation studies [62,63]. Furthermore, this study demonstrated the value of using theoretical frameworks at multiple levels of the system (e.g., organizational, individual) to facilitate behavior change with the health care environment. Moreover, this study, as part of a larger parent study, both engaged consumers and conducted formative and development research activities to acknowledge the context and other implementation factors that may influence whether providers and clinics adopt and utilize this intervention [33,64].

In conclusion, findings suggest that an evidence- and theory-based eHealth app could serve as an innovative mechanism to assist providers (NPs and midwives) in translating the prenatal oral health guidelines into clinical practice. These findings further support the utility of theory-based interventions in facilitating provider behavior change and improving quality of care using innovative technology.

Compliance with Ethical Standards

Funding: The study was funded by the National Institute of Dental & Craniofacial Research of the National Institutes of Health under award number R41DE023723. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Conflict of Interest: Cheryl A. Vamos, Stacey B. Griner, Claire Kirchharr, Shana M. Green, Rita DeBate, Ellen Daley, Rocio Quinonez, Kim A. Boggess, Tom Jacobs, and Steve Christiansen declare that they have no conflicts of interest.

Authors’ Contributions: CV, RD, ED, TJ, and SC conceptualized the study; CV, SBG, SMG, RD, ED, RW, and KB developed instruments; CV, SBG, and SMG collected data; CV, SBG, SMG, and CK analyzed data. All authors contributed to manuscript preparation and editing.

Ethical Approval: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1,964 Helsinki declaration and its later amendments or comparable ethical standards. This study is approved by the University of South Florida’s Institutional Review Board (Pro00018122). This article does not contain any studies with animals performed by any of the authors.

Informed Consent: Informed consent was obtained from all individual participants included in the study.