Development of a Novel Computerized Clinical Decision Support System to Improve Adolescent Sexual Health Care Provision

Abstract

Objective:

To develop an acceptable clinical decision support (CDS) system to facilitate evidence-based sexual health care for adolescents in the emergency department (ED).

Methods:

In this multi-phased iterative process, we engaged an expert group to synthesize evidence on a wide range of sexual health services (e.g., contraception, condoms, identification and treatment of previously diagnosed sexually transmitted infection). We created a computerized questionnaire and embedded our decision-tree, utilizing patient-entered responses to create tailored, evidence-based recommendations, and embedded links to study-related resources such as the emergency contraception (EC) Quick Guide. We utilized mixed methodology to explore perspectives of adolescents aged 14–19 years and clinicians at two general and two pediatric EDs after they interacted with the system. Clinicians reported usefulness (Likert scale 1=not at all, 4=very); adolescents reported acceptability. We used the Chi-square test to compare responses between subgroups. We collected adolescents’ verbatim responses to open-ended questions; clinicians self-entered responses. Four authors independently generated themes from qualitative responses before compiling key findings and achieving consensus on final themes.

Results:

Among 57 clinicians (23 physicians, 23 nurses, 11 NPs; 54% female; 65% aged <40 years), the mean system usefulness rating was 3.4 ± 0.7. Gender, age, clinician role, or ED type were not associated with rating the system “somewhat/very” useful. Clinicians identified barriers (e.g., time constraints) that could be overcome by implementation considerations (e.g., training) as well as benefits including improved care. For future assessments, providers preferred computer (65%) over face-to-face interview (26%). Among 57 adolescents (mean age 16.2 years; 75% female; 56% sexually experienced), nearly all (95%) reported it was “very/somewhat easy” to complete the computerized questionnaire and to understand the questions. Most adolescents understood the EC Quick Guide and correctly identified that ulipristal, compared to levonorgestrel, required a prescription and was more effective. For future assessments, adolescents preferred computer (69%) over face-to-face interviews (9%).

Conclusions:

We developed a sexual health CDS system that is easy to use and can facilitate evidence-based care to reduce health outcome gaps. Evaluation of system impact on service delivery and, ultimately health outcomes, is needed.

Sexually transmitted infections (STIs), human immunodeficiency virus (HIV) transmission, and unintended pregnancy are costly public health problems that disproportionately affect U.S. adolescents.¹ Among adolescents in the emergency department (ED), rates of infection with chlamydia and gonorrhea are up to 7 times higher than the general population and the pregnancy risk index (PRI), an estimate of pregnancy risk in the subsequent 12 months, for females in two EDs was more than three times greater than the national average.^2–5 While appropriate care can prevent these outcomes, many adolescents do not receive high-quality sexual healthcare due to adolescents’ limited care seeking as well as provider- and system-level barriers.^6–8 Limited access to care, due to insurance and geographic barriers, contributes to reliance on safety net systems, including emergency departments (EDs). Adolescents make almost 19 million ED visits each year, frequently for sexual health complaints.⁹ Despite commonly presenting with complaints potentially related to the genitourinary system, adolescents often do not receive evidence-based care.^10–11 Many ED clinicians, as well as those practicing in other settings, report time constraints and knowledge deficits as barriers to providing this specialized type of care.^12–13 While the ED represents an important, but underutilized location for reaching large numbers of at-risk adolescents, few proven interventions are available in the ED to reduce health outcome gaps by addressing patient, provider, and system-level barriers.

Interventions utilizing computerized systems can improve care and have been successfully used to target a wide range of provider behaviors (e.g., vaccine provision, medical documentation).¹⁴ Computerized clinical decision support (CDS) systems utilize software to link patient characteristics with the best possible medical knowledge base and then generate patient-specific assessments or recommendations for clinicians.¹⁵ Computerized CDS systems are a scalable strategy for health promotion and are effective at overcoming ED-specific barriers such as provider time constraints and knowledge gaps.^16–17

Recently, computerized systems were used in two EDs to improve rates of testing for STIs/HIV and Goyal et al. also developed a brief screening tool that can be used to identify adolescents at risk for STIs.^18–20 While promising, these systems were limited in scope (e.g., focused on STI testing) and did not adequately address provider-level barriers such as knowledge deficits. Thus, their use alone is unlikely to lead to sustained reductions in rates of STIs/HIV and unintended pregnancy. To achieve optimal health, adolescent ED patients also need counseling on safer sex practices, links to comprehensive care, and access to condoms and contraception, in addition to evidence-based testing.²¹

Medical innovation can be challenging to implement into practice. Involving end users and key stakeholders to inform the development of novel systems can facilitate successful development and implementation of these models in clinical settings.²² This study is a component of a multidisciplinary research project which aims to develop, implement, and measure the efficacy of a brief behavioral intervention to improve adolescent access to sexual health services and reduce sexual risk behaviors. The objective of this work is to describe the iterative process of developing a novel CDS system that is acceptable to adolescents and providers and facilitates care for the wide range of sexual health needs (e.g., contraception, identification and treatment of previously diagnosed STI, condoms and risk reduction counseling) for adolescents seeking care in the ED.

OVERVIEW

We employed a multi-phased, iterative process to develop an evidence-based, computerized CDS system to support the wide range of sexual health needs among adolescents in the ED (Figure 1). We solicited provider and adolescent feedback on the system via mixed-methodology. The institutional review boards for each participating site approved this work.

Figure 1.

Iterative process to develop our CDS system. CDS = clinical decision support; HIV = human immunodeficiency virus; PED = pediatric emergency department.

METHODS

CDS System Development and Expansion

Development:

In 2014, we engaged a multidisciplinary team with expertise in health services research, epidemiology, adolescent sexual health and psychology, evidence-based medicine, and emergency medicine to guide initial development of our CDS system. Team members reviewed existing evidence and published guidelines focusing on adolescent services that could be provided easily by non-expert providers in an acute care setting: testing for pregnancy and STIs (i.e., Chlamydia trachomatis and Neisseria gonorrhoeae), pregnancy prophylaxis with emergency contraception (EC), counseling for safer sex behaviors, treatment for previously diagnosed patient–reported STIs, and condom provision. We utilized an institutional evidence-based practice team that used Cochrane Collaborative software packages (i.e., Review Manager, Gradepro) and a validated tool (i.e., Appraisal of Guidelines, Research and Evaluation [AGREE] instrument) to assess existing guidelines and develop recommendations for services missing from the literature.^23–24 The recommendations are well established for some services (e.g., annual Chlamydia screening for sexually active adolescent females), while others are less clear (e.g., annual Chlamydia screening for heterosexual adolescent males). We describe an example of a clinical question, the corresponding search strategy and selected articles in Data Supplement S1 (available as supporting information in the online version of this paper). Our team achieved consensus by reviewing the evidence and discussing risks and benefits, costs, applicability, and expert opinion.

Once all service recommendations were finalized, we created a computerized decision tree to determine appropriate services based upon risk behaviors (Figure 2). We created a health questionnaire, based on previously validated national surveys, to assess adolescent sexual risk behaviors.^25–27 We then embedded the decision tree within a software program (e.g., Research Electronic Data Capture [REDCap]) to create a branch-logic questionnaire, which utilized patient-entered responses to create tailored service recommendations (e.g., treatment for previously diagnosed STI, emergency contraception, Figure 3). All tailored recommendations were then displayed on a final computerized screen.

Figure 2.

Emergency contraception (EC) algorithm.

Figure 3.

Example participants. GC/CT = chlamydia/gonorrhea; HIV = human immunodeficiency virus.

Because of well-documented patient and provider-level barriers to EC use and provision as well as emerging evidence regarding differential efficacy of medications, much of our work in this phase focused on EC .^{7, 28–30} Thus, we created and embedded links to an EC quick guide (Figure 4) for provider and adolescent use and links to relevant EC scientific publications.^31–34

Figure 4.

Emergency contraception quick guide. BMI = body mass index; LMP = last menstrual period.

Expansion:

In 2016, we re-engaged our multidisciplinary team to guide expansion of our CDS system to include additional health services (i.e., testing for human immunodeficiency virus (HIV) and some quick-start methods for contraception). We again reviewed evidence for previously incorporated health services including evidence published since our last review regarding EC efficacy.^35–38 We used the same processes described above to guide these efforts. Due to the increasing complexity and number of health services, we also ranked each health service to determine prioritization order for service recommendation, considering time-sensitivity (i.e., treatment for previously diagnosed STI) as well as individual and public health relevance. We also embedded contraception links for provider and adolescent use (i.e., birth control choices chart, quickstart algorithm, contraindications).^39–40

System evaluation

Design:

We conducted a cross-sectional survey of ED clinicians as well as adolescents seeking care in the ED to explore acceptability and ease of use of the computerized CDS system and to identify perspectives on system usefulness.

Setting:

We conducted this work at four urban EDs within academic, tertiary-care hospitals; three (two general and one pediatric EDs) were located in the Midwest and one (pediatric ED) in the Northeast (Table 1). The pediatric EDs were located within free-standing pediatric hospitals and staffed by at least one pediatrician who is fellowship-trained in pediatric emergency medicine (or currently in fellowship). Additional staff included physicians who completed residency in emergency medicine or pediatrics. The two general EDs were staffed by at least one physician who was residency-trained in emergency medicine. In all EDs, nurse practitioners cared for patients independently and nurses specialized in acute care or pediatric acute care. Resident physicians and nursing trainees also provide care at each ED.

Table 1.

Characteristics of Participating Emergency Departments

	Pediatric 1	Pediatric 2	General 1	General 2
Annual patient volume	70,508	91,868	64,025	55,000
	N (%)	N (%)	N (%)	N (%)
Adolescents (14–21 years)	9,320 (13.2)	13,025 (14.2)	4,362 (6.8)	5,010
Race/Ethnicity
Black	26,260 (37.3)	55035 (59.9)	34,662 (54.1)	14135 (25.7)
White	22,799 (32.4)	22493 (24.5)	20,769 (32.4)	28930 (52.6)
Hispanic	14,168 (20.2)	6852 (7.5)	4,676 (7.3)	7480 (13.6)
Other/unknown	7281 (1.1)	7488 (8.1)	8,594 (13.4)	4455 (8.1)
Payor Status
Medicaid/Medicare	49,210 (69.8)	59,356 (64.6)	26,311 (41.1)	9061 (16.5)
Commercial	15,081 (21.4)	29,378 (32.0)	7,542 (11.8)	6237 (11.3)
Self-pay	6,147 (8.7)	2,630 (2.9)	27,671 (43.2)	2214 (4.0)
Unknown/other	70 (0.10)	504 (0.5)	25,01 (3.9)	37488 (68.1)

Participants and Recruitment

Adolescents:

Adolescents aged 14–19 years seeking care at a single Midwestern pediatric ED were eligible. Subjects were excluded if they did not speak English, had significant impairment that would impede participation as determined by the ED provider (e.g., severe illness, developmental delay, intoxication), had complaints involving sexual assault or psychiatric issues, or were wards of the state. Trained research assistants (RAs) used the electronic tracking system to identify potential participants. We used purposive sampling to ensure characteristics of particular subgroups of interest (i.e., sexually experienced). We obtained a convenience sample across a wide range of hours (generally 8:00am-12:00pm each day), based upon research assistant (RA) availability to recruit. Written parental consent was waived because this study conferred minimal risk and requiring consent has been shown to contribute to bias.⁴¹ Participants provided verbal consent/assent and there was no incentive for participation. Adolescent and clinician participants were recruited in two phases; some evaluated the original CDS system (2014–2015), others evaluated the expanded CDS system (2016–2017).

Clinicians:

Full-and part-time nurses, nurse practitioners, and physicians (including fellows training in pediatric emergency medicine) who staffed the participating EDs were eligible. We used email and verbal announcements to recruit participants during when available during a shift or at a scheduled time outside of care delivery. We used purposive sampling to include diverse perspectives (e.g., clinical role, gender, length of time since training). Participants provided verbal consent; there was no incentive for participation. To focus on consistent ED staff and to increase participant homogeneity, we excluded resident physicians and nurses in training.

Measures:

A multidisciplinary team developed the assessment, based in part on pertinent literature. Our survey was pilot tested for ease of use; no significant issues were identified with the final version, which took about 15 minutes to complete. For all participants, we assessed demographics and preferences for future sexual health assessments (i.e., computer, face-to-face interview, written, no preference).¹⁹ Among providers, we assessed work experience, number of adolescents in clinical practice, attitudes towards CDS systems and adolescent care (e.g., comfort with contraceptive counseling), and system usefulness (single item).^{28, 42} Among adolescents, we assessed worry about health information privacy when using a computer (1= not at all, 5=extremely) and perceptions on CDS system (i.e., ease of use, length, and comprehension).¹⁹ For all participants recruited during the second phase, we added the system usability scale consisting of 10 Likert-scale items (clinicians) or a single adjective rating scale (adolescents).^43–45 The open-ended questions assessed: 1) facilitators and barriers to CDS system use, recommendations for EC quick guide and CDS system modifications (among adolescents), and 2) perceptions on system benefits, facilitators and barriers to system use as well as reproductive care provision, and recommendations for EC quick guide and CDS system modifications (among providers).

System Demonstration and Data Collection: Research assistants (RAs)—Based on our prior decade of work, our RAs are experienced in conducting sexual health research and knowledgeable of diverse sexual health topics. For this study, RAs received 2 hours of training in study protocol including how to access and demonstrate the CDS system. Adolescents—To ensure exposure to the EC quick guide, participants first observed the RA demonstrate the CDS system. Assuming the hypothetical identity of an adolescent female, the RA completed the computerized questionnaire and reported unprotected intercourse within the past five days. This generated a service recommendation to discuss and offer EC which allowed the RA to highlight the EC quick guide. Participants then interacted with the CDS system by independently entering actual or hypothetical behavior data and reviewed additional tailored service recommendations with the RA. Participants completed a computerized survey about the system, directly entering their responses. To facilitate efficiency, RAs read the open-ended questions and typed adolescent responses (verbatim) into the computer. Clinicians—Participants first observed the RA demonstrate the CDS system, again assuming the hypothetical identity of an adolescent female with recent unprotected intercourse. This generated a service recommendation to discuss EC, allowing the RA to highlight the EC quick guide and also the web-link to relevant literature. Participants then interacted with the CDS system by independently entering hypothetical behavior data and reviewed additional tailored service recommendations with the RA. Participants then completed a computerized survey about the CDS system. For open-ended questions, clinicians directly entered their responses with RAs available for assistance if needed. Throughout data collection, the team met regularly to discuss study progress, recruitment, and quality control issues. Study data were collected and managed using REDCap electronic data capture tools hosted at the lead institution.⁴⁶

Data Analysis

Quantitative:

Demographic characteristics were summarized using standard descriptive statistics. For the single item “How useful is a system like this in the ED?”, provider responses were dichotomized into useful (“very” or “somewhat”) and not useful (“a little useful” and “not useful at all”). We calculated the system usability score according to published methods.^43–45 We used the Chi-square test to compare bivariate responses between subgroups (e.g., ED clinical role) and the Spearman’s rank-order correlation to determine the strength and direction of association between provider ratings of system usefulness and attitudinal variables (e.g., comfort with adolescent sexual health). We considered p<0.05 as statistically significant. We used SPSS Statistics 22.0 (SPSS Inc, Chicago, IL, USA) for all analyses.

Qualitative:

Qualitative responses were formatted into codes determined a priori. Analysis team members (MKM, PCI, AP, CJM) read all transcripts and generated themes independently. One author summarized key findings into analytic memo, which was shared and discussed. Each member then independently reread the transcripts to assess agreement with the identified key findings. Members then resolved any discrepancies and achieved consensus on final themes.

RESULTS

Acceptability and Usability

Clinicians:

Participants (n=57) were enrolled from November 2014-January 2015 and from August 2016-March 2017 (Table 2). Compared to the general ED, pediatric ED clinicians were more likely to report treating ≥ 11 adolescents each month (39% vs. 79%, p =0.002). Similar percentages of eligible nurses (PED 4.6%, general ED 4.6%) and physicians (PED 18.5%, general ED 20.3%) participated from each setting.

Table 2.

Provider participant characteristics

	All (N=57)	Pediatric 1 (n=15)	General 1 (n=14)	Pediatric 2 (n=14)	General 2 (n=14)
Clinical Role
Physician	23 (40)	6 (40)	6 (43)	6 (43)	5 (36)
Nurse	23 (40)	6 (40)	5 (36)	6 (43)	6 (43)
NP/PA	11 (20)	3 (20)	3 (21)	2 (14)	3 (21)
Female	31 (54)	10 (67)	5 (36)	8 (57)	8 (57)
≤ 10 years ED experience	40 (70)	9 (60)	11 (79)	9 (64)	11 (79)
≥ 11 adolescents per month	34 (60)	11 (73)	7 (50)	12 (86)	4 (28)
≤ 40 years of age	37 (65)	9 (60)	11 (79)	11 (79)	10 (71)
Very/somewhat comfortable providing adolescent sexual healthcare	42 (74)	10 (67)	11 (79)	12 (86)	9 (64)

Overall, ED clinicians reported high CDS system acceptability and usability (Table 3). Gender, age (< 40 years vs. ≥ 40 years), provider type (nurse vs. other), ED type (pediatric vs. general), and comfort with sexual health (“very/somewhat comfortable” vs. “neutral/somewhat/very uncomfortable”) were not associated with rating the system “somewhat/very” useful (all p>0.05). More favorable ratings of system usefulness were associated with 1) more favorable ratings of CDS systems in general (.35, p=0.008); 2) more agreement with asking about sexual health (.41, p=0.002), offering condoms (.30, p=0.025) or testing for STIs/pregnancy (.38, p=0.004) during an ED visit; and 3) interest in a system to help the participant (.58, p<0.001) or someone else provide this care (.50, p<0.001). Among ED providers (n=27) who completed the System Usability Scale, the mean score was 73.7 ± 12 (range 47.5–100), which translates to “good” on 7-point adjective scale (worst to best imaginable). For future assessments about sexual health behaviors, providers preferred computer (65%) over face-to-face interview (26%), while 5% had no preference. These factors were not associated with preference for future sexual health assessment: gender, age (less than 40 vs. 40 and older), provider type (nurse vs. other), ED type (pediatric vs. general), and comfort with sexual health (“very/somewhat comfortable” vs. “neutral/somewhat/very uncomfortable”) (all p>0.05).

Table 3.

Acceptability and usability assessment

Emergency Department (ED) Providers (n=57)
	Number (%) responding
How useful is a system like this in the ED?	50 (88) “Very/somewhat useful”
How interested are you in having a system like this in your ED to help you provide sexual health care for adolescents?	37 (65) “Very/somewhat interested”
How interested are you in having a system like this in your ED to help someone else provide sexual health care for adolescents?	47 (82) “Very/somewhat interested”
How did you feel about the amount of time it will take to provide this type of care to an adolescent in the ED?	19 (33) “Way too/Somewhat long”
Adolescents (n=57)
How was the amount of time it took to complete these questions about your health?	38 (67) “Somewhat/very short”
How easy or difficult it was to take this questionnaire on the computer?	54 (95) “Very/somewhat easy”
How easy or difficult was it to understand the questions?	54 (95) “Very/somewhat easy”
How worried were you about others finding out your health information when answering these questions on a computer?	47 (87) “Not at all/slightly worried”

Adolescents:

Adolescents were recruited November 2014-January 2015 and August 2016-November 2016. Fifty-seven participated (67% of approached; mean age 16.2 years; 75% female; 56% sexually experienced). Overall, adolescents reported high CDS system acceptability and usability (Table 3). Among a subset of participants (n=31), 58% rated the system as “best imaginable” or “excellent.” For future sexual health assessments, adolescents preferred computer (69%) over face-to-face interview (9%), while 21% had no preference. Participants who were “not at all/slightly” worried about their health information were more likely to have no preference for future assessment, compared to those “somewhat/moderately/extremely” worried (38% vs.4%, p=0.013).

EC Quick Guide

Adolescent responses regarding interpretation of the EC Quick Guide are shown in Table 4. Correctly identifying which medication worked better overall (ulipristal) and on days 4–5 (ulipristal) and required a prescription (ulipristal) was not associated with gender, younger age, or reporting previous sexual intercourse. Among ED providers, 91% found the EC quick guide “very/somewhat helpful.”

Table 4.

Adolescent interpretation of EC Quick Guide

Looking at the EC Quick Guide,…	Number (%) responding (N=57)
…which medicine may work better at preventing pregnancy overall?*
Ella (Ulipristal)	39 (69)
Plan B (Levonorgestrel)	8 (14)
They are about the same	10 (18)
…which medicine may work better at preventing pregnancy if the unprotected sex was 4–5 days ago?*
Ella (Ulipristal)	44 (77)
Plan B (Levonorgestrel)	4 (7)
They are about the same	6 (23)
…you need a prescription to buy which medicine? *
Ella (Ulipristal)	47 (83)
Plan B (Levonorgestrel)	2 (3)
Both	6 (11)
Neither	2 (3)

^*Desired response=ulipristal

Qualitative Data—Adolescents

Most adolescents seemed to understand the EC Quick Guide and that ulipristal, compared to levonorgestrel, required a prescription and was more effective on days 4–5 following unprotected sex. Generally, adolescents liked the chart describing birth control options – although one person didn’t like the picture of the needle. Some commented on how much information was available and how it was easy to understand. However, a few thought there was too much information and could be improved by shortening, using bullets, or discussing with a provider. The vast majority thought the chart would be very helpful when talking to the provider about starting or changing birth control.

Overall, adolescents were satisfied with the computerized survey. A few suggested we should explain why the questions are being asked, talk about confidentiality, and define some of the terms and the contraception charts more clearly. Some suggested additional questions asking about healthy relationships or consensual intercourse. A few indicated they would want to get health services and/or talk with someone about sex during their visit. One person reported preference for speaking to someone over using an iPad. Most teens thought the language used was clear, although some reported specific terms (e.g, oral sex, spermicide) could be difficult to understand. While some said teens prefer “slang” or simpler language, others felt the more “medical” terms were appropriate given the setting.

Most adolescents felt getting follow-up care is important and would not be difficult to complete. Facilitators to completing follow-up were having been to clinic previously, supportive parents (e.g., mother) and having access to transportation. Some felt missing school and needing to involve a parent were barriers to follow-up.

Qualitative Data—Clinicians

Providers felt that the benefits of the system include speed/efficiency, increased disclosure from the adolescent, and the ability to provide personalized information/education in a standardized way that can ultimately improve health care delivery as well as overall health (Table 5). Frequently cited potential barriers were: limited time/ED flow constraints, challenges in maintaining privacy, and technological concerns (e.g., computer theft, interfacing with electronic health record). Several had concerns that visit time will be taken up addressing issues that were not related to the chief complaint, leading to increased resource utilization, increased length of stay, and potentially misuse of the ED setting as a site of care. Respondents also had concerns about patients not answering honestly and challenges due to limited health literacy and/or English proficiency. Finally, several felt that personal values of the staff (e.g., moral or religious conflicts) could impact utilization of the system (Table 5).

Table 5.

Illustrative Quotes from Providers

Theme	Illustrative Quotes
Perceived Barriers	“Not an ED physician’s responsibility to provide emergency contraception for every patient having sex.”
	“A small portion of this goes against some of my personal beliefs, although the benefit is there as well for a large majority of this system being used in our patient population.”
CDS System can work in the ED setting	“Ideally screening could be integrated with regular clinical care using devices available within each patient room. Positive responses could cue clinicians…to order testing, provide treatment, involve other resources such as health educators,…and link them to resources available in the community.”
Perceived Benefits	“This has the potential to identify patients at risk who are uncomfortable providing honest answers face-to-face. It also defines various types of sexual intercourse that may not be comfortably (or completely) differentiated by some providers in the ED setting. It is thorough, and the same questions would be asked to each patient without concern for missing anything.”
	“Time efficient. Easy to read summary page.”

Participants recommended procedures that could “create a culture” where this type of work could be facilitated and where families “understand the expectation” of confidential interviews. Suggestions included standardizing the approach, including signs and routinely asking parents to leave the room; fitting into workflow between triage and provider evaluation, standing orders and easy access to follow-up appointments, extensive staff training and/or specialized staff to administer the tool, and streamlined electronic health systems. Two respondents mentioned providing an “opt-out for certain aspects of the program.”

Most providers seemed to think a system such as this could work in the ED, in particular if staff are well trained, if the project is supported by non-physicians, and if workflow and follow-up issues are addressed (Table 5). Some, however, felt this would not work or should only be used on subset of patients (e.g., those with genitourinary complaints). Many providers commented that although integration would likely meet resistance at first, people will “come around eventually.”

Most providers believed nurses and physicians can use the system. Some felt it would be better for the RN to administer the survey; some felt any member of the team including front-line ordering providers could administer the survey. Most also felt the ED is an appropriate setting to use this tool. Providers also mentioned primary care and other outpatient settings including obstetrics/gynecology clinic and urgent care; a few also mentioned inpatient settings. Suggestions for additional questions included partner violence / unwanted sexual encounters, safety, substance use/abuse, testing for additional STIs, and a free text option for the teen to write in any specific questions.

Providers seemed to be able to easily understand the EC Quick Guide. However, seven providers expressed concerns or questions about EC. A few providers wanted to know how often a female patient might actually need EC for recent unprotected sex while others had specific questions about the mechanism of ulipristal and whether it could cause abortion or damage a fetus.

Concerns around Quickstart contraception included privacy concerns, starting a chronic medication, provider knowledge around contraception, interfering with the PCP-patient relationship, lack of time to counsel appropriately and lack of adequate follow-up. To reduce these barriers, providers mention close linkage with the PCP, better training / dedicated training of ED staff (particularly RNs), and easy access to refills. Concerns around HIV testing were similar to those in the literature – inadequate time for counseling, inadequate linkage to care, limited access to rapid test, maintaining patient privacy, and patient fear of testing.

DISCUSSION

We describe the comprehensive, iterative process we employed for developing and evaluating a novel, evidence-based CDS system to generate tailored sexual health service recommendations for adolescents seen in the ED. Nearly all adolescents felt that the questions were easy to understand and using the computerized system was easy. The vast majority of ED and PED providers felt the system was useful and could be thoughtfully integrated into workflow. This innovative system goes beyond recent ED-based efforts to improve STI care by supporting a wide range of adolescent sexual health needs, including pregnancy prevention and sexual risk reduction practices (i.e., condoms) and by incorporating our novel EC Quick Guide as well as web-based links for both providers and adolescents to facilitate discussion on more complex topics (i.e., contraception). Further, our system generates a prioritized list of tailored services to target time sensitive care (e.g., EC, treatment for existing STI) which is especially important in the busy ED setting. Prioritizing the services also facilitates delivery of patient-centered care by helping the adolescent conceptualize risk, which can be challenging during this developmental stage. Regarding emergency contraception, ED clinicians could limit their discussion of this medication to females at immediate risk (e.g., higher priority) or provide education to all sexually-active adolescents which is still evidence-based, though lower priority.

While we utilized a research assistant to guarantee that all participants were exposed to certain system features (e.g., emergency contraception for recent unprotected intercourse), our CDS system can be used independently by adolescents. The printed or electronic tailored recommendations could then be presented to individual clinicians to act upon as indicated. This is similar to recent efforts targeting adolescent STIs and mental health, where ED providers received screening results and decision support regarding potential next steps.^{18–19, 47} Alternatively, to streamline and standardize the process of identifying and responding to individual sexual risk (which can vary widely), these tailored recommendations could be incorporated as part of a larger behavioral intervention to improve health outcomes.

Interestingly, favorable ratings of system usefulness were not associated with gender, age, provider or ED type, or comfort with sexual health. In contrast, some studies of EC, violence prevention programming, and sickle cell disease care have demonstrated differences between nurses and physicians as well as general and pediatric ED providers, with nurses and general ED physicians reporting more negative attitudes and/or practices.^7,48–50 Because many providers (26%) lacked comfort with adolescent sexual health and most preferred computers for these health assessments, our CDS system could facilitate improved care by standardizing collection of risk behaviors and supporting provision of evidence-based health services. Of note, more respondents were interested in having a system like this to help someone else provide sexual health care, compared to using the system themselves (82% vs. 67%), which may be related to lack of comfort with adolescent sexual healthcare as described in the literature.^7,13,28

Although providers identified specific barriers to implementing this CDS system, most agreed that these barriers could be overcome. Further, many acknowledged the system could be useful in improving health outcomes, ultimately translating into reduced health disparities. The vast majority of ED visits for children are to non-specialty facilities (i.e., general, non-academic). Practice differences in use of laboratory testing and imaging, adherence to clinical practice guidelines, and hospitalization rates are associated with both ED physician training (pediatric vs. general) as well as ED type and location (e.g., academic, non-metropolitan).^51–55 CDS systems can reduce variability in care and may be especially beneficial in ED settings where clinicians lack expertise and comfort with adolescent sexual health care provision.^14–17 Further, our CDS system can be customized to provide local referral information, for services not available at the index visit.

Many providers suggested that education and consideration of work flow before and during implementation could facilitate programmatic success over time. Use of an implementation framework, such as the consolidated framework for implementation research, can provide structure for thoughtful consideration of these contextual factors, leading to improved implementation and translation into meaningful patient health outcomes.⁵⁶ Additionally, successful models for implementing computerized systems to intervene with alcohol, suicide, and violence prevention in ED settings may also provide implementation guidance for individual EDs.^{47, 57–58}

In response to specific participant suggestions or concerns, we added questions about number of partners, modified the system to clarify language, and elaborated on the confidentiality of responses to our final version. Some adolescents demonstrated inadequate understanding of the EC quick guide and some providers asked for clarification on mechanism of action; further efforts could focus on improving clarity.

Limitations

There are some limitations to this study. We recruited providers from four large EDs but limited adolescent recruitment to a single ED due to staffing and budget constraints. This limits the generalizability of our results though we successfully utilized purposive sampling techniques to ensure inclusion of specific subgroups. Participant responses were subject to social desirability bias. To counter this, we used experienced RAs who received additional trained in data collection. The RAs handled all aspects of study recruitment and study authors were not aware of which providers enrolled. And although our sampling wasn’t driven by a goal for thematic saturation, our analysis demonstrated clear congruence around themes.

Conclusions

In summary, we developed a computerized CDS system that incorporates the best available science to facilitate personalized, prioritized sexual and reproductive care for adolescents in the busy ED setting. This easy-to-use system could be universally applied to ensure that all adolescents receive evidence-based care, regardless of clinician expertise. Further research is needed to evaluate the efficacy of the system to increase health service delivery and ultimately improve health outcomes.