Primary care clinicians’ perspectives on clinical decision support to enhance outcomes of online obesity treatment in primary care: A qualitative formative evaluation

Abstract

Objective:

Online behavioral treatment for obesity produces clinically-meaningful weight losses among many primary care patients. However, some patients experience poor outcomes (i.e., failure to enroll post-referral, poor weight loss, or premature disengagement). This study sought to understand primary care clinicians’ perceived utility of a clinical decision support system (CDSS) that would alert clinicians to patients’ risk for poor outcome and guide clinician-delivered rescue interventions to reduce risk.

Methods:

Qualitative formative evaluation was conducted in the context of an ongoing pragmatic clinical trial implementing online obesity treatment in primary care. Interviews were conducted with 14 nurse care managers (NCMs) overseeing patients’ online obesity treatment. Interviews inquired about the potential utility of CDSS in primary care, desired alert frequency/format, and priorities for alert types (non-enrollment, poor weight loss, and/or early disengagement). We used matrix analysis to generate common themes across interviews.

Results:

Nearly all NCMs viewed CDSS as potentially helpful in clinical practice. Alerts for patients at risk for disengagement were of highest priority, though all alert types were generally viewed as desirable. Regarding frequency and delivery mode of patient alerts, NCMs wanted to balance the need for prompt patient intervention with minimizing clinician burden. Concerns about CDSS emerged, including insufficient time to respond promptly and adequately to alerts and the need to involve other support staff for patients requiring ongoing rescue intervention.

Conclusions:

NCMs view CDSS for online obesity treatment as potentially feasible and clinically useful. For optimal implementation in primary care, CDSS must minimize clinician burden and facilitate collaborative care.

Obesity is a major public health concern, affecting an estimated 40% of adults in the United States (Hales et al., 2017). Defined by a body mass index (BMI) ≥ 30 kg/m², obesity is associated with increased risk for several chronic health conditions, including cardiovascular disease and type 2 diabetes mellitus (Abdullah et al., 2010; Poirier et al., 2006). Intensive behavioral weight loss treatment is recommended as a first-line treatment for obesity, as it routinely produces clinically meaningful weight losses and improves cardiovascular functioning and blood glucose regulation (Look AHEAD Research Group, 2010; Wing et al., 2011). Primary care is an ideal setting for behavioral obesity treatment delivery given providers’ contact with a large portion of the US population across demographic groups (Fitzpatrick et al., 2016; Jensen et al., 2014). However, numerous barriers impede delivery of in-person treatment in primary care, including insufficient time, lack of training, and reimbursement issues (Antognoli et al., 2017; Luo et al., 2021; Yarnall et al., 2003). To address barriers, behavioral obesity treatment is increasingly delivered online (Little et al., 2016; Thomas et al., 2015) and is effective for primary care patients (Bennett et al., 2018; Thomas et al., 2015). These programs are typically semi- or fully-automated to reduce burden on primary care clinicians; patients access comprehensive weight loss resources without requiring counseling from physicians or other medical staff.

Although online obesity treatment has strong empirical support, many patients experience poor outcomes due to: failure to enroll after referral; premature disengagement from the program; and poor weight loss (i.e., insufficient for health benefit) despite continued program use (Ahern et al., 2016; Neve et al., 2010; Unick et al., 2017). Evidence suggests that delivery of brief “rescue interventions” (e.g., phone coaching, motivational interviewing) may improve obesity treatment outcomes (Dennison et al., 2014; Unick et al., 2016), but no infrastructure currently exists in routine clinical practice to inform primary care clinicians about which patients need intervention and when.

Across many domains of medical care, clinical decision support systems (CDSS) help clinicians filter large amounts of patient health data and interpret their meaning to guide evidence-based patient care decisions (Hunt et al., 1998; Kawamoto et al., 2005). In the domain of obesity care, CDSS has primarily been tested as a strategy to improve physician documentation of obesity in electronic medical records (Gangadhar et al., 2018; Lee et al., 2009). These systems typically compute a patient’s body mass index automatically at each patient encounter and provide an alert when the result is in the range of overweight or obesity, prompting a physician to document the concern in the patient’s problem list. Though less well studied, some prior work has also explored the effects of more extensive CDSS systems that provide guidance on care planning for obesity. Results indicate that such programs can improve comprehensiveness of follow-up care plans for weight management in adults (Bakken et al., 2014) and increase referral to appropriate adjunctive clinical care among children in the pediatric setting (Shaikh et al., 2014). CDSS could also be used to guide targeted, effective, and efficient rescue interventions for adult patients undergoing obesity treatment in primary care, including online interventions. One large trial to date has explored the utility of adjunctive ongoing nursing support to facilitate adult patient engagement with an online obesity treatment platform and found that it produced favorable weight loss outcomes (Little et al., 2016). However, that study involved regular face-to-face or phone contact between nurses and all enrolled patients and only offered additional “rescue” support when patients had already begun to regain weight. The utility of CDSS for anticipatory rescue interventions remains untested. We are aware of no CDSS developed to date that allows clinicians to monitor patient progress during obesity treatment, and none have been developed which are designed to anticipate and prevent risk for suboptimal outcomes before they occur. Furthermore, little is known about primary care clinicians’ views on CDSS for obesity as an adjunct to online intervention. Thus, there is a need to better understand whether such a CDSS could be useful in routine care settings, and how it should be designed.

The present study sought to investigate primary care nurses’ perspectives on the utility of a CDSS focused on online obesity treatment and their logistical preferences for integrating it into routine clinical practice. Given that nurses are often the primary care clinicians tasked with supporting online or app-based obesity interventions (Critchlow et al., 2020; Little et al., 2016), they were identified here as the target end users of CDSS. The study capitalized on an opportunity to interview nurse care managers (NCMs) who were already using an existing, entirely automated online obesity treatment—Rx Weight Loss (RxWL)—across a statewide primary care practice network, supported by an implementation-effectiveness pragmatic clinical trial (R18 DK114715; Espel-Huynh et al., 2019). Patient recruitment for the trial is completed and follow-up assessment is ongoing. Consistent with prior research on technology-facilitated obesity interventions (Serrano et al., 2016), NCMs anecdotally observed that some patients experience premature disengagement or difficulty losing weight. Thus, we hypothesized that some patients may be in need of clinical outreach for rescue intervention, which could be facilitated in the future by a novel CDSS.

The aim of this study was to gather necessary stakeholder information to inform the development and future implementation of a CDSS designed to facilitate rescue interventions for online obesity treatment. To this end, we conducted a developmental formative evaluation, which occurs during the first stage of an implementation project (Stetler et al., 2006). Developmental formative evaluation seeks to enhance the likelihood of success in the particular setting of a project. It involves collection of data on potential influences on future implementation, such as determinants of current practice, potential barriers and facilitators to practice change, and to adoption of an evidence-based practice or innovation. Data from a developmental formative evaluation enable researchers to understand potential problems and, where possible, overcome them prior to initiation of interventions in study sites through identification and use of robust implementation strategies (Stetler et al., 2006). Developmental formative evaluation in this study consisted of a qualitative inquiry of NCMs’ views regarding a hypothetical CDSS that would facilitate rescue interventions for RxWL patients at risk for poor outcomes. The aims of the formative evaluation interviews were twofold: (1) explore clinicians’ interest in CDSS; and 2) understand facilitators and barriers to implementation of an online obesity treatment program in primary care. This manuscript summarizes only content related to the CDSS-focused aim.

Methods

Routine Care Setting & Context

Although this developmental formative evaluation focused on future CDSS implementation, the existing RxWL clinician-facing platform used in the current trial is described briefly here for context. NCMs served two or more practices within the Rhode Island Primary Care Physicians Corporation (RIPCPC), a network of approximately 60 primary care clinics across the state of Rhode Island. All RIPCPC clinics follow a Patient-Centered Medical Home care model (Lipson et al., 2012). As such, NCMs’ roles included managing care for patients with complex medical concerns, including obesity, cardiovascular disease, and type 2 diabetes mellitus. The implementation setting and randomized trial are detailed further in (Espel-Huynh et al., 2019). All NCMs were able to view patient enrollment via a secure online clinician dashboard linked to the RxWL program. In the larger trial, approximately half of NCMs were assigned to a Basic implementation condition; their dashboards included a simple list of patients enrolled in RxWL. The dashboard for remaining NCMs in the Enhanced condition included all Basic features, plus information on individual patient progress (e.g., total weight change, last login, etc.), but RxWL did not provide any guidance on how to interpret or act upon the patient data NCMs viewed. For the purposes of this qualitative formative evaluation, we determined that combining data for the Basic and Enhanced groups was most appropriate (see “Data Processing and Analysis” below).

Participants

NCM participation for both the larger trial and the qualitative study was optional and voluntary. Participants were recruited via verbal announcement at monthly implementation meetings between the research team and NCMs. Interested NCMs were instructed to inform their nursing manager if they wanted to participate. They were assured that their choice to opt in or decline participation would not affect their employment or involvement in the larger trial. All NCMs employed by RIPCPC had previously consented to participate in the larger trial. Of those employed at the time of the qualitative study (N = 14), all agreed to be interviewed.

Interviews (each approximately 1-hour in duration) were primarily conducted on-site at RIPCPC headquarters; one NCM was located in a rural setting and was interviewed in her primary care practice. NCMs were informed that the choice to participate (or decline) would not be shared with RIPCPC and would not affect participation in the larger trial. Prior to interviews, NCMs completed written informed consent and a brief demographic questionnaire. Interviews were conducted individually in a private room and audio recorded. NCMs received $60 for participating. All procedures were approved by the Institutional Review Board of the corresponding author’s home institution.

Interview Content

Interview questions were generated by the qualitative team lead and revised with other team members’ input. Questions were modified slightly after each of the first two participant interviews to improve clarity and ensure that questions addressed relevant content. The semi-structured CDSS portion of the interview (see Supplemental Materials) began with a brief description of a hypothetical CDSS to be used in conjunction with the existing RxWL clinician-facing dashboard. As described above, depending on assigned condition in the current trial, the existing dashboard allowed clinicians to view a list of which patients were enrolled (Basic), plus information on patient engagement and progress (Enhanced). NCMs were informed that the CDSS would use a computer algorithm to predict individual risk and alert them of patients likely to experience any of the following: (1) failure to enroll post-referral to RxWL; (2) poor weight loss outcomes; and (3) premature program disengagement. NCMs were then asked about their perceptions of the potential helpfulness of CDSS for RxWL, as well as preferred alert frequency and delivery mode. Details on the clinical interventions that may be used in response to these alerts was intentionally left vague by the interviewer. This allowed NCMs to suggest the possible approaches most appropriate for their setting and scope of practice.

To inform future design of the most efficient CDSS possible, NCMs were also asked to rank-order the alert types from most to least useful in clinical practice. This would allow us to narrow our future CDSS design efforts to include stakeholders’ most highly-prioritized alert types. NCMs were only asked to rank-order the alerts they viewed as potentially useful; if a NCM did not view CDSS as helpful overall, she was not asked to give any rankings. The semi-structured interview guide is available to interested readers in the Supplemental Materials.

Researcher Characteristics

The multidisciplinary research team included individuals with expertise in behavioral obesity treatment, technology-based interventions for health behavior change, implementation science, and kinesiology. Two members had specific prior training and practical experience with qualitative methods for technology development. Researcher biases and assumptions were discussed throughout all stages of coding and analysis to maintain reflexivity.

Data Processing and Analysis

This manuscript follows guidelines outlined in the Standards for Reporting of Qualitative Research (O’Brien et al., 2014). Data were analyzed using a rapid qualitative analytic approach (Hamilton, 2013; Hamilton et al., 2019), which is commonly used to yield comprehensive yet targeted qualitative results on a briefer timeline (e.g., Taylor et al., 2018). The intent is to facilitate more rapid translation to the target setting.

Recorded interviews were transcribed with identifiers removed, reviewed for transcription errors by a second research team member, and coded using a matrix analysis approach (Averill, 2002; Hamilton, 2013). Only portions pertaining to CDSS were analyzed for the present study. Transcript data from all fourteen NCMs was included in analyses; prior research indicates that 5–10 interviews per stakeholder type is typically sufficient to reach saturation (Guest et al., 2006; Palinkas et al., 2015). Consistent with a rapid qualitative approach (Hamilton, 2013; Hamilton et al., 2019), a transcript summary template was co-developed by the two coders (HE and OF) and approved by the team’s auditor (CG; team member with specialized training in qualitative methods). All qualitative data pertaining to CDSS in each respective NCM interview was summarized by both coders in separate transcript summary sheets, then transferred to a final consensus summary after team members discussed and resolved discrepancies. Consensus transcript summaries were then transferred to a matrix for coding, organized by NCM and trial implementation condition. Initial review of emergent themes from NCMs using the Basic versus Enhanced versions of the RxWL dashboard indicated that nurses had similar perspectives on CDSS regardless of dashboard type, thus results are aggregated across all NCMs for this study.

Domains and codes were generated by the primary coders (HE and OF) and iteratively adjusted to best reflect interviewees’ perspectives as coding progressed. For example, if a transcript included a phenomenon similar to that described in previously coded transcripts, but elaborated in some way such that the current category name did not adequately capture the idea expressed, the category was updated to encompass this new data and all prior data captured under the same category. NCM rankings for CDSS alert priorities (non-enrollment, poor weight loss, and/or early disengagement) were listed in the matrix and viewed across all participants. Overall priority was assigned based the total number of NCMs who expressed any interest in each respective alert type, then by their individual rankings (e.g., the alert type ranked most frequently as the highest-priority alert). All coding decisions were finalized using a consensus approach, such that any initial disagreements were resolved collaboratively. Such instances were rare and typically involved minor discrepancies. For example, in one instance, an NCM initially indicated that she would find the alerts helpful, but as she discussed her perspective further, she described several dimensions of ambivalence. Initially, this was coded by one coder as viewing CDS favorably, while the other coded for ambivalence and concerns about CDS (although both coders had coded sections of her interview under the “Concerns” domain listed in Table 1). After further discussion, both coders agreed that her ambivalence was critical to capture and coded her overall perspective as such.

Table 1.

Emergent Domains, Categories, and Illustrative Quotes from Matrix Analysis

Domain	Category	Detailed Description	Illustrative Quote from Participant
CDSS for RxWL is viewed as useful	Helps NCM address barriers with patient	Alerts would provide an opportunity for NCMs to reach out, explore issues, and help patients overcome current barriers to program engagement and/or success.	Maybe the people who are reluctant to keep going. So the challenge…is having to write down everything that they eat. But it is a necessary thing too. They really need to do it in order to see what they’re—that’s the big challenge they have, they think it’s a lot of work….time consuming.
	Helps NCM track patient progress and/or hold them accountable	Currently, NCMs have limited means by which to track patient progress aside from their own clinical judgment. Since NCMs viewed accountability as key for many patients, more information on patient progress was viewed as a tool to support patients more closely.	You usually get a good feel for it, but not always. Ones that you think they’re going to go home and sign up, they end up not doing it.
	Guides motivation enhancement intervention	Motivational interviewing is commonly used by NCMs to facilitate a wide range of patient behavior change processes. Many NCMs indicated that the alerts could prompt them to use this intervention technique in the context of the online obesity treatment.	If they’re not losing weight, everyone loves to see results…you’d want to reach out and be like, “Hey…keep working on it. This is normal.” You know, going through that, and, ”Just keep striving,” and yeah. I think they would be good notifications.
	Could facilitate coordinated care efforts to support weight loss	For patients who would benefit from more comprehensive support than the NCM can provide, the alert could direct NCMs to refer to clinicians with complementary skill sets, such as wellness coach, dietitian, behavioral health, etc.	RIPCPC is developing a wellness program, so it might be something good to turn into something that the wellness program would make the outreaches too. The wellness program is more of the continuous touch, “Hi. How are you doing,” thing.
Email alerts are the most preferred method of CDSS delivery	Email (most commonly preferred)	Many NCMs preferred email alerts, because they frequently use secure email to communicate about clinical care throughout the day.	We check it all the time.
	Rx Weight Loss platform (preferred by some)	Some NCMs believed they logged in to RxWL frequently enough that the alerts would be seen promptly if delivered there. Delivery in the RxWL platform would reduce email burden, which was desirable for some.	For us the RxWL platform is something separate we have to log in. And you know, maybe I would probably log in twice a week. And I think that way, you’re not constantly getting something popping up in your email. I think that would probably work really well for us as a whole.
	Electronic medical record (least preferred)	Though less common, some NCMs thought that EMR alert delivery would be least burdensome and increase likelihood of being seen and acted upon.	I’m thinking if it could be in the medical record, it would be something that we’re already in. Because we have to sign in and out of SharePoint, so we wouldn’t necessarily be going in there. We’d probably miss it, not be right on the ball with it.
Alert frequency & timing should balance need for prompt patient care with other clinical duties	CDSS alert should come 1-week post-enrollment	--	Maybe a week later so it’s not so far out and it’s still fresh on their mind, but not harassing.
	CDSS alert should be delivered once per week	--	You don’t want to wait too long because then they're gone.
	CDSS alert should be delivered every 2 weeks	--	If you’re checking on those patients once a month or every couple weeks, it would be okay. More than that, I’d get so many other emails.
	Number of CDSS alerts per patient should be limited	--	But maybe just once, because if someone doesn’t want to continue the program, I don’t want to be like a telemarketer and make them—force them. Because then you lose that… relationship, you know?
Adaptations may be needed to align with the primary care setting	Use streamlined patient data interpretation	NCMs indicated that alerts needed to provide a quick and easy way to see which patients were in need of support, and similarly a way to notate that a rescue intervention had been delivered.	[In RxWL,] You could have a little list of the people maybe that have not signed in so we could click them off one at a time as we go.
	Provide alerts for positive outcomes (e.g., weight loss)	Many NCMs preferred to focus on positive outcomes in addition to preventing negatives, thus they were also interested in being notified of patient successes such as weight loss milestones, successful program completion, etc.	Instead of focusing on the negative, I’d rather focus on the positive. I’d like to know if they’ve lost a lot of weight, so I can call them and give them encouragement. “Great job! Keep going!”
	Align CDSS with clinician’s scope of practice	NCMs indicated that some patients may require rescue interventions that were beyond their scope of practice.	And it would be really really helpful to have, like, support of reaching out to these patients who have never logged in, just to alleviate my role…because that’s not really a nursing role, per se.
	Include other care team members in patient support efforts	Given patients’ potential need for additional support, the CDSS should ideally include options for referral to other clinicians and support staff.	Most of these health issues for obesity have to do with behavioral health. I mean, a lot of people overeat because of anxiety or whatever it is. So that’s maybe a missing piece, the behavioral health component with the weight loss…you could have a virtual coach—you know? Or something like that.
Concerns, including excessive clinical burden, must be considered	Potential for excessive clinician burden	Across most interviewees, a primary concern was that CDSS alerts would increase clinical burden to an unmanageable level. To address this concern, NCMs expressed a need for CDSS to involve only brief, one-time interventions. One NCM requested that patient outcomes not be tied to NCM performance evaluation in any way.	I don’t want to have to be where it puts me accountable as if I’m a failure if those people don’t [enroll]. You know, because sometimes we’ve had feedback on other programs in the past with a lot of pressure, okay? And when—there’s always so much you can push people to do, patients to do. Like I said, this is just a little piece of their whole office visit and care plan. I’m trying to get them to exercise, I’m trying to get them to drink more water.
	Questionable effectiveness of NCM outreach to patients	Given the constraints of NCMs’ time and scope of practice, a few NCMs expressed concern that one-time outreach would be insufficient to “rescue” patients from suboptimal outcome.	If they’re struggling with weight loss, I mean, I can certainly talk with them and see what’s going on. But I don’t know how effective that will be.
	Potential to produce preconceived, negative notions about patients	Though rare, one concern was that alerts about poor outcomes may reduce optimism about patient progress in a way that could be detrimental to the patient-clinician relationship.	You don’t want to look at a patient in a negative light. Like I don’t want the computer saying, “Oh, she’s not a follow-through person,” and then she might have been totally open to me. You know, I don’t want to give her an unfair chance.

Upon completion, the analytic coding matrix was then reviewed by the auditor (CG) and auditing feedback integrated by the coding team before all analysis decisions were finalized. Themes from the data were briefly and concisely summarized in line with recommendations for rapid qualitative analysis (Hamilton, 2013; Hamilton et al., 2019).

Trustworthiness and Credibility

Trustworthiness and credibility of results were ensured via consensus coding of all transcripts. The research team held ongoing discussions of researcher biases (e.g., checking team members’ favorable assumptions about CDSS in primary care) and ways to mitigate bias during coding (e.g., including an essential coding domain to capture concerns and perceived negative aspects of CDSS). Final coding decisions were reviewed by an independent auditor who was not involved in conducting interviews or initial coding. Based on auditor feedback, the coding team added codes for missed phenomena and consolidated redundant categories.

Results

A summary of emerging themes is described below. Illustrative quotes are displayed in Table 1. Interviewee characteristics are summarized in Table 2. All NCMs self-identified as female, White, and Non-Hispanic and had attained at least a 2-year college degree/certificate.

Table 2.

Participant Demographic Characteristics

Variable	N (%)
Gender
Female	14 (100%)
Mean Age (range)	51.50 (25 – 64)
Race
White	14 (100%)
Ethnicity
Non-Hispanic	14 (100%)
Educational attainment
2- year college degree or certificate	6 (42.9%)
Bachelors degree	7 (50.0%)
Masters degree	1 (7.1%)
Personal technology devices used
Smartphone device	14 (100%)
Laptop or desktop PC	13 (92.8%)
Tablet PC	10 (71.4%)
Other internet-connected device (Smart TV)	1 (7.1 %)
Rx Weight Loss implementation condition
Basic	6 (42.9%)
Enhanced	8 (57.1%)

CDSS for RxWL is viewed as useful, particularly to prevent disengagement

Nearly all nurse care managers believed that CDSS would be helpful in supporting patients’ success in the RxWL program. NCMs generally reported that current contact with and tracking of patients post-enrollment was limited. They believed a future CDSS may help them monitor patient progress more closely and identify those who are struggling. In response to the proposed CDSS alerts, NCMs stated that they would likely explore barriers a patient is facing and intervene to address them. NCMs were already trained in motivational interviewing; many wanted to employ motivation enhancement interventions for patients struggling to implement the behavior changes recommended by the RxWL program (e.g., daily self-monitoring of food intake, exercise). Some NCMs also suggested that alerts may facilitate coordinated care efforts to support patient weight loss. For example, NCM outreach in response to an alert may reinforce physician medical recommendations given at the time of RxWL referral. If patients need ongoing support to stay engaged with RxWL, a CDSS alert could prompt referral to non-NCM resources, such as RIPCPC’s more intensive Wellness Program—which provides a fitness tracker and regular nutrition visits with a RIPCPC clinician—or other behavioral health services.

When asked to rank the three types of CDSS (non-enrollment, poor weight loss, and/or early disengagement) in terms of priority, NCMs tended to place the highest priority on alerts for disengagement. NCMs believed their intervention skill set (e.g., motivational interviewing, problem-solving) aligned well with these patients’ needs and that outreach efforts may be more fruitful for individuals who had already engaged (versus those not enrolled). NCMs highlighted several barriers to engagement that they have observed anecdotally among their RxWL patients (e.g., viewing self-monitoring as “time-consuming”). The second-priority alert was for less-than-expected weight loss. NCMs believed such alerts could prompt outreach to identify areas of low adherence (e.g., not meeting calorie or exercise goals) and employ problem-solving. The primary concern about this alert was that the NCM’s brief outreach may be insufficient to improve program adherence and subsequent weight loss. Non-enrollment alerts were more controversial. NCMs generally expressed a higher level of confidence in their ability to anticipate non-enrollment and follow up on it without the help of an alert reminder, thus many viewed it as unnecessary. However, some NCMs noted that this alert may become more useful as the number of enrolled RxWL patients increased with more sustained program implementation.

Email alerts are the most preferred method of CDSS delivery

Among NCMs who were interested in receiving alerts, email was the most preferred delivery format versus other types (e.g., through the RxWL platform or in a patient’s electronic medical record). One major perceived benefit of email alerts was its integration into existing workflows; NCMs reported checking email frequently throughout the day and were more confident they would see an email alert in a timely manner. However, this was not the case for all NCMs, as one indicated that she receives so many emails per day that an alert could be easily overlooked. A suggested alternative—particularly for NCMs who were already accessing the clinician dashboard at least weekly—was to have alerts delivered immediately upon login to the RxWL platform. The electronic medical record (EMR) was an unworkable alert venue for several NCMs since not all of their primary care practices used the same EMR system.

Alert frequency and timing should balance need for prompt patient care with other clinical duties

When asked about the ideal frequency and timing with which CDSS alerts should be delivered, NCMs expressed an understanding that alerts must be frequent enough to allow swift follow-up with patients who may be at risk of poor outcomes (i.e., before they become entirely disengaged and the intervention becomes irrelevant). However, receiving alerts too frequently for a given patient would be excessively burdensome for NCMs. Furthermore, unnecessarily frequent outreach could lead patients to feel like NCMs are “hounding” them. For non-enrollment in particular, NCMs requested a single alert that would be delivered approximately one week after the referral. However, some NCMs viewed this as unnecessary, since they already had their own strategies to follow-up with patients who had not yet enrolled.

Preferred frequency for alerts of risk for poor weight outcomes or program disengagement differed from non-enrollment. NCMs generally preferred to receive an alert report once per week, listing all patients who may require timely outreach, along with any patient-specific data that could inform intervention (see Adaptations section below for more detail). Anything more often than weekly was viewed as burdensome, especially considering NCMs’ numerous other work responsibilities. Two NCMs preferred to receive alerts every two weeks or monthly. However, these two NCMs were willing to receive alerts more often if it were necessary to prevent poor patient outcomes.

Adaptations may be needed to align with the primary care setting

Many NCMs suggested that the CDSS be optimized for primary care. For example, NCMs wanted the information on why a patient is struggling to be streamlined in the RxWL platform (e.g., by highlighting names of at-risk patients in red, along with information on current weight loss, exercise, and calorie intake). Those NCMs who were already logging in to RxWL frequently suggested that these adaptations could replace email alerts in many cases. NCMs also requested that the CDSS focus on patient successes (e.g., achieving a weight loss goal) in addition to risk for poor outcomes. This would allow NCMs to reinforce progress as well as support patients through their challenges. Finally, NCMs emphasized the importance of ensuring that the suggested patient outreach fits within their scope of practice. They reported that a single outreach for motivational work or problem-solving would be within the bounds of a NCM’s duties, but anything beyond that would require additional support from other care team members.

Concerns, including excessive clinical burden, must be considered

NCMs expressed some concerns about CDSS, most commonly potential excessive clinician time burden. NCMs described several competing work responsibilities and feared that excessive alerts may interfere with those tasks. Though less common, NCMs also raised concerns about how effective a one-time outreach would be—especially for patients who are not losing weight. Finally, one NCM described a previous negative experience with a patient outreach system that was tied to her work performance evaluations, and this added pressure for her. She preferred that her response to the RxWL CDSS alerts (and the patient’s outcome) remain separate from work performance measures.

Discussion

Nurse care managers in primary care viewed CDSS as a potentially useful tool for managing and preventing poor patient outcomes in online obesity treatment, particularly if the CDSS was designed to target premature disengagement from the program. Conducting a developmental formative evaluation within the context of the RxWL study allowed us to understand that future, successful implementation of CDSS would require appropriate alert frequency that minimizes NCM burden (at most once weekly) and promotes prompt patient outreach. Some concerns arose, including those of excessive clinical burden, pressure for performance if patients did not respond to CDSS-prompted interventions, and skepticism that one-time outreach would be sufficient to improve a patient’s progress in online obesity treatment. In cases where ongoing support would be required (i.e., more than one contact with a patient), an integrated or collaborative care model would be needed to facilitate patient success.

CDSS alerts for early disengagement emerged as a high priority for NCMs. This is striking, given that in the context of the larger RxWL trial, at least half of NCMs (Enhanced group) had access to patient engagement data that could allow them to identify which patients have already become disengaged. Further, NCMs reported that they are already trained in some clinical interventions they might use to re-engage these patients (e.g., motivational interviewing). Despite this, they reported doing limited outreach with the current system (no CDSS). Empirical evidence suggests that clinicians tend to be overly optimistic about their patients’ adherence to medical recommendations (Clyne et al., 2016) and tend to have difficulty detecting negative patient outcomes (e.g., treatment nonresponse or dropout) without CDSS alerts and clinical support tools to guide rescue intervention—even when raw outcome data are available (Shimokawa et al., 2010). This same tendency toward optimism may be present for NCMs using RxWL. In this case, a CDSS could prove most valuable if it focuses on disengagement and also includes tools to guide clinicians in using their existing clinical skills to address barriers, enhance motivation, and keep patients engaged.

Results align with prior qualitative findings of a need for electronic health care interventions to fit closely with the needs of the end users in a clinical setting (van Gemert-Pijnen et al., 2011). NCMs’ requests for streamlined data interpretation in the RxWL platform align with a large body of literature indicating that successful CDSS deliver patient health data in a visually-appealing and abbreviated format with concrete suggestions for clinician intervention (e.g., Foraker et al., 2015). If a future CDSS can accommodate these needs, it may improve clinicians’ responsiveness to patient challenges during weight loss, maintain treatment efficiency, and minimize burden for NCMs; clinicians’ outreach efforts would specifically target patients in need. Overall, these results have practical relevance for implementation research, health technology, and primary care service delivery. They provide insight into CDSS components that can be successfully integrated into primary care, particularly within large practice networks and those following a Patient-Centered Medical Home model (Lipson et al., 2012).

Some characteristics of this study limit its generalizability. All participants came from a single statewide practice network in Rhode Island. CDSS may be viewed differently by practitioners in other states or in other countries with different healthcare infrastructures. In addition, although all NCMs who were eligible agreed to participate, the sample had limited diversity in terms of gender, race, and ethnicity. Notably, though, these NCMs serve primary care practices that vary by organizational size, patient demographics and clinical needs, and geographical distribution (i.e., rural versus urban settings), and thus their clinical perspectives and experiences are diverse. Finally, while our intent was to understand NCMs’ perspectives on CDSS for obesity treatment, independent of our own views as researchers, it is impossible to remove researcher bias entirely from qualitative work. We took deliberate steps to minimize the influence of these biases on the results, including self-assessment and discussion of biases, consensus coding, and second-audit at multiple stages during data collection and analysis.

Conclusions & Future Directions

Successful implementation is dependent on seeking stakeholder input, and formative evaluation is the method for doing just that. Results from this study may inform future work in designing, developing and testing a CDSS in conjunction with online obesity treatment in primary care. A top priority of CDSS alerts is for patients at risk of premature disengagement. In addition to notifying clinicians of a patient’s risk status, clinicians also desire information about how they might intervene to help. Technical CDSS design should account for a range of NCM alert preferences, perhaps with customizable data view options within the CDSS platform (e.g., turn email alerts on or off). Accommodating different user needs may increase acceptability. Collaborative care interventions may also be required to supplement NCM outreach to patients. Thus, future work should explore how CDSS can accommodate varying collaborative care models across clinics (i.e., depending on differential availability of support staff), and could also consider whether supplemental, technology-mediated rescue interventions (e.g., automated text support or online motivation enhancement programs) may suffice in settings with more limited support staff available.