Feasibility of using an artificially intelligent chatbot to increase access to information and sexual and reproductive health services

Abstract

Background

Following the US Supreme Court decision overturning Roe v. Wade, there is evidence of limitations in access to safe abortion care. Artificially intelligent (AI)-enabled conversational chatbots are becoming an appealing option to support access to care, but generative AI systems can misinform and hallucinate and risk reinforcing problematic bias and stigma related to sexual and reproductive healthcare.

Method

A single arm pilot study describing the development of an AI chatbot focused on sexual and reproductive health and its deployment in a clinic setting and community-based organization over a nine-month period.

Results

We adjusted chatbot content based on feedback from the medical director and clients of organizations where the system was deployed given updated medical guidelines and preferred language related to gender-affirming care. We deployed the system in two organizations and tracked use over nine months. In that time, there were 1749 queries from 425 unique users. One-tenth of users of the clinic based chatbot went on to schedule an appointment for care.

Conclusions

Ongoing challenges in accessing sexual and reproductive health suggest having diverse mechanisms to facilitate access to accurate and updated medical information is warranted. Using an AI chatbot is feasible to accomplish this goal and shows promise in increasing opportunities to access care.

Introduction

People face challenges in accessing sexual and reproductive health (SRH) and may experience shame and stigma when they access care. The US Supreme Court decision overturning Roe v. Wade reduced access to safe abortion care. ¹ Restrictions on gender-affirming care are increasing, ² issues that are amplified for racial and ethnic minority populations in the USA, where disparities in sexually transmitted infection (STI), safe abortion and pregnancy outcomes remain intractable. ³

Technologies such as social media, mobile applications and text messaging can effectively improve SRH.^4,5 Artificially intelligent (AI)-enabled conversational chatbots are an appealing option to support access to care.^6–10 These systems go beyond fixed state a priori text message libraries pushing out predetermined content users must select (e.g. text “1” if you need information on medication abortion). Artificially intelligent–assisted chatbots allow users to initiate and self-direct questions that systems interpret and classify using generalized language models. Models are trained using a large corpus of text and fine-tuned to accurately classify text. ¹¹

Generative chatbots such as ChatGPT use these same processes and are trained on language models inclusive of all content available on the Internet, but they can offer misinformation when asked about health-related topics, including SRH.^12,13 The content of message responses when engaging with a chatbot is also important,^14–16 and we have research detailing effective strategies to increase message engagement and maximize effects by using strategies like tailoring, timely “nudges” for decision-making, stories and positive emotion and skills building.^17–21 Messaging must also avoid reinforcing stigma ²² and should reduce bias. ²³ We have yet to effectively apply these strategies to AI-assisted chatbots in health and integrate them in meaningful ways with healthcare delivery systems. This brief communication documents the feasibility of using an AI-assisted chatbot within clinic and community organizations to facilitate access to information about and care related to SRH.

Method

We implemented a program evaluation to assess the feasibility of using an AI-assisted chatbot to deliver SRH information using the People at the Center for Mobile Application Design (PACMAD) model. ²⁴ This model assesses system usability (frequency of engagement and use, delivery of system content without error) and accuracy as well as system acceptability and practicality for use in clinic settings without disruption. We partnered with staff in leadership positions in two organizations, both of which desired to implement a chatbot on their respective websites with information about SRH. One partner was a health clinic located in Boulder, Colorado, offering SRH services. Our primary contact in this organization was the medical director, with a medical degree and specialization in primary care whose responsibility was to design and oversee the delivery of all SRH education-related initiatives for the organization. Our second partner was the Colorado Black Health Collaborative, a grassroots, nonprofit, community-based organization (CBO) focused on raising awareness of and supporting interventions to address disparities in health outcomes for Black, African American and African people. Our primary contact in this organization was the executive director, who has a doctoral degree in business and healthcare administration and specific expertise both in health disparities for African Americans and in community engagement for health-related programs among urban African Americans (Figure 1).

Figure 1.

Presentation of chatbot on healthcare website.

With the help from our partner organizations, we categorized specific topics we believed people wanted to learn about SRH. We grouped topics into domains: (a) contraception; (b) safe abortion; (c) STIs; (d) gender-affirming care; (e) laws and access to SRH care; and (f) specialized gynecologic care. We identified 290 topics across these six domains that we believed were relevant after reviewing frequently asked questions about SRH listed on the websites of the Centers for Disease Control and Prevention (CDC), the World Health Organization (WHO) and the Guttmacher Institute. We then created a pool with 25 or more variations on ways to ask questions related to each topic so the system could be “trained” to offer the correct response based on an inference related to what the user asked. For example, one user may ask, “What are the side effects of using birth control?” While another might inquire, “What symptoms can I expect when I take the pill?” Both queries would be matched to the “hormonal contraceptive side effects” topic.

Using a natural language processing and machine learning (ML) workflow, responses were offered to match the intent behind user questions. When the system could not match the user input to a question, it replied with a “pick list” such as the following: “I think you are asking about one of these topics: (a) what is an STI, (b) What infections are sexually transmitted, (c) What treatments can I get for an STI? Please type the letter corresponding to the topic you wish to explore or try your question again.” We sought to correctly match the user input question to the intent of the question 80% of the time, aligned with industry standards that offer an expected threshold for accuracy. ²⁵

Because of the risk of breaches to confidentiality using text messaging, content was only available on websites, and we encrypted IP addresses to ensure user anonymity. We developed a curated library of response messages in English and Spanish guided by the strategies for message design mentioned above. Our library of responses addressed the key constructs proposed by social and behavioral change theories, including increasing knowledge and awareness of sexual health domains, forming positive attitudes and beliefs to healthy sexual behaviors, understanding social norms and enhancing self-efficacy. ²⁶ We also validated the content by cross-checking at least two credible sources (e.g. CDC, WHO, and FDA) to ensure information accuracy.

Our clinical partner reviewed the content for medical accuracy. The CBO partner invited 14 of their clients to review messages and comment on how well messages resonated, whether they were easy to understand and if they offered useful information. Sample response messages are demonstrated in Table 1.

Table 1.

Examples of theory-based chatbot responses to promote sexual health.

Intent	English	Spanish	Theoretical construct
What are the different types of birth control?	You have many birth control options to choose from. Some methods are ones you get at the clinic like an IUD or implant that last 3 years or more. Some are ones you can get every 3 months (the depo shot), every month (the ring,) every week (the patch), or every day (the pill). Other methods must be used every time someone has sex that could lead to pregnancy. Some methods are permanent, like a vasectomy or tubal ligation, so people who never want to be pregnant may choose one of these	Tiene muchas opciones de anticonceptivos de donde elegir. Algunos métodos son los que obtienes en la clínica como un DIU (Dispositivo intrauterino) o implante que duran 3 años o más. Algunos son los que puedes obtener cada 3 meses (anticonceptivo inyectable de DEPO), cada mes (el anillo) cada semana (el parche) o cada día (la píldora). Se deben utilizar otros métodos cada vez que alguien tenga relaciones sexuales que puedan conducir al embarazo. Algunos métodos son permanentes, como una vasectomía o ligadura de trompas, por lo que las personas que nunca quieren estar embarazadas pueden elegir uno de estos	Self-efficacy
What is a medical abortion?	Medical abortion is when you take pills to terminate a pregnancy. You may have heard it called “the abortion pill,” and over half of the people who had abortions in 2020 chose this method. It involves taking pills at home over a couple of days and is best if done within the first 10 weeks of pregnancy. In this state, you can meet with a doctor via phone or video chat and can even get the abortion pills in the mail	El aborto médico es cuando toma pastillas para terminar un embarazo. Es posible que lo haya escuchado llamado “la píldora abortiva” y más de la mitad de las personas que tuvieron abortos en el 2020 eligieron este método. Este procedimiento involucra tomarse pastillas en su casa durante un par de días y es mejor si se hace dentro de las primeras 10 semanas de embarazo. En este estado, puede consultar con un médico por teléfono o chat de video e incluso puede obtener las píldoras de aborto por correo	Knowledge and awareness
What is a bacterial STI?	Bacterial STI include chlamydia, gonorrhea, and syphilis. In most cases, if you have one of these STI, you will have no symptoms, but sometimes, people get unusual discharge—maybe it smells or itches—or pain when peeing or have sores or bumps in the genital area. If you have symptoms or not, if you think you may have an STI, a provider can test you and get you treatment	Las ITS bacterianas incluyen clamidia, gonorrea y sífilis. En la mayoría de los casos, si tiene uno de estos ITS, no tendrá síntomas, pero a veces las personas obtienen una descarga inusual, tal vez huele o pica, o dolor al orinar o tienen llagas o bultos en el área genital. Si tiene síntomas o no, si cree que puede tener una ITS, un proveedor médico puede examinarlo y obtener su tratamiento	Knowledge and awareness
What is gender-affirming care?	Laws in this state recognize and support gender preferences and the seeking of gender-affirming care. This includes mental healthcare and support for male-to-female or female-to-male transition and/or care and treatment needed after transition. Studies show that one of every four people who are transgender, nonbinary, or gender diverse has put off seeking medical care because they fear or have experienced not being treated with respect. This could create negative impacts on their health and well-being. Link here if you are seeking more information about how to access gender-affirming care in Colorado	Las leyes de Colorado reconocen y apoyan las preferencias de género y la búsqueda de atención para afirmación de género. Esto incluye atención de salud mental y apoyo para la transición de hombre a mujer o mujer a hombre, y/o cuidados y tratamientos necesarios después de la transición. Los estudios muestran que 1 de cada 4 personas que son transgénero, no binarias o diversas de género han pospuesto la búsqueda de atención médica porque temen o han experimentado no ser tratades con respeto. Esto podría crear impactos negativos en su salud y bienestar. Este es un recurso si busca más información sobre cómo acceder a la atención de afirmación de género en Colorado	Norms, positive attitudes

This work was considered program evaluation and was exempted from institutional review board review by the Colorado Multiple Institutional Review Board. This exemption indicates that the IRB does not consider this to be research, and we were not required to obtain informed consent from people who reviewed messages or who interacted with the chatbot system.

We collected data regarding system usage, accuracy, and user clicks to make an appointment for care. Usage was reflected by the total number of users, the total number of questions posted to the chatbot, and the average number of questions asked for each user. We logged each question posed to the system and each response sent by the system as well as time of day and date for each interaction. We assessed accuracy, that is, the percentage of chatbot responses that correctly answered users’ questions either by providing a direct response or a pick list. Mismatched queries were reviewed and reclassified manually to improve the ML model. If there was no existing topic to match a query, and the query was related to one of our domains, we would create new topics, develop theory-informed responses and generate variations on queries.

Users of the system embedded on our clinic partner's website could click on a link included with every chatbot response that would take them directly to the clinic appointment scheduling page. We tracked the total number of system users and the number of link clicks to calculate the proportion of people using the system who went on to seek care.

Results

After developing an initial message library, we shared the content with our clinic and community partners as described above. They requested multiple changes to medical content; for example, information about breast self-examination has evolved and this is no longer recommended practice; the medical director from our clinic partner suggested language aligned with current guidelines for breast self-awareness. She also advocated that we remove gendered references as these might reinforce gender dysphoria for gender nonconforming clients, advocating for more neutral language to say “people,” instead of saying “women” or “men” in messages.

The 14 clients of our community partner included 12 female and 2 male participants. The majority (i.e. 10) were aged 35 and younger, and four were older than 35. They made suggestions via two recorded Zoom sessions that generally corroborated the suggestions from our clinic partner. However, we observed divergences shown in Table 2 where there was disagreement in what content to present and how to present it across our partner organizations.

Table 2.

Areas where the healthcare organization and CBO partner disagreed about content for the SRH chatbot.

Issue/consideration	Healthcare partner	CBO partner
Message content re: social needs	Did not mention this as a concern	Requested greater emphasis, e.g. offer detail about how limited transportation might affect ability to get to a health appointment, along with links to bus routes and discounted fare information for public transportation
Message content re: health disparities	Did not mention this as a concern	Requested information that women of color face higher rates of morbidity and mortality related to breast and cervical cancer than other groups because of delays in screening and/or diagnosis and/or limited access to care along with links to opportunities for free screening events in the community
Presentation of gender when discussing care	Advocated gender-neutral terms for all messaging, e.g. saying “pregnant people” instead of “pregnant women” or “people with a uterus” instead of “women with a uterus”	Indicated the use of gender-neutral terms was not universally accepted among the African American community and endorsed the use of “man”, “woman” and “person” interchangeably

CBO: community-based organization; SRH: sexual and reproductive health.

Both the leadership and clients of The Colorado Black Health Collaborative indicated that ongoing mistrust of public health and healthcare delivery among members of the Black, African American, African, and other communities of color meant that the chatbot would likely not be accessed and utilized unless there was an explicit endorsement from trusted community leaders and that having the system embedded on their websites implied that the content was more trustworthy and credible than information found elsewhere online.

Based on this feedback, we revised and finalized our message library prior to deployment. To address discrepancies in preferences for presentation of gender-affirming care content, we alternated labels for “women,” “men,” and “people” across messages. We worked to embed the system with organizational logos on partner websites to ensure credibility.

The system was available as a conversational agent embedded on each partner organization's website from 23 June 2023. We report on use over nine months. Table 3 illustrates the PACMAD feasibility items we documented.

Table 3.

PACMAD feasibility outcomes, SRH AI chatbot.

Frequency of engagement	1749 queries from 425 users; mean of 4 queries per user
Delivery of system content without error	328/1749 responses reclassified (19%)
Precision	1382/1749 (79%)
Practicality for use in clinical settings	10% of users on clinic site self-scheduled an appointment

AI: artificially intelligent; PACMAD: People at the Center for Mobile Application Design; SRH: sexual and reproductive health.

In addition to the detail in Table 3, we generated new content, including referrals to local resources (e.g. where to screen for HPV), followed by access to care questions (e.g. eligibility for insurance). Users of the chatbot on the clinic site most often asked about the cost of an abortion, followed by queries related to abortion legality. The next most popular topic is related to scheduling an appointment. Users of the chatbot on our CBO partner site most often asked questions about abortion, contraception and whether one can get sexual or reproductive healthcare as a minor in our state.

Discussion

We report on outcomes related to a pilot implementation of a SRH, website-based, AI-assisted chatbot for a clinical care provider and for a CBO. We demonstrate that we can create an AI-assisted chatbot with curated content related to SRH that is medically accurate and conforms to evidence-based strategies for message design. System use over time shows it can facilitate access to information and allows people to gather information prior to making an appointment for care. The system functions as intended.

Automating access to SRH information may reduce barriers that impeded access to care, particularly treatment, and SRH resources can improve the scale, efficiency and equitable access to SRH care. Artificial intelligence–assisted chatbots driven by ML models are increasingly common, but are susceptible to misinformation and hallucination and are not commonly integrated with healthcare scheduling systems and/or the electronic health record to improve access to and reimbursement mechanisms for care.^12,13,27,28

Brief digital interventions can be delivered effectively via text messaging to positively impact health behaviors and health outcomes, including outcomes related to SRH.^15,29 We can adapt these interventions and deliver them via AI-assisted chatbot as a next generation of technology-based health communication initiatives. ³⁰ Although we have no evidence to date that using AI-assisted chatbots specifically can impact health behavior in general or SRH, we do have evidence that fixed state chatbots are effective, with recent trials demonstrating they can be used effectively to improve chronic illness medication refills in the short term. ³¹ This work underscores opportunities to generate more sophisticated technologies in support of SRH that can overcome emergent challenges with AI.

Limitations

The advent of generative AI systems suggests we can reach and engage many thousands more people than those described in this pilot project, and our small number of participants does not leverage the full capacity of AI to anticipate a full range of queries. As a project evaluation with no comparison groups, we cannot determine if this system is superior to text messaging or other technologies to share information or improve access to care; although not our objective, we also cannot assess if the system improved SRH knowledge.

Conclusions

Our work shows that using AI-assisted chatbots to facilitate access to sexual and reproductive healthcare is feasible and acceptable, which can be beneficial for healthcare organizations. The system described here can be adapted for other health-related topics. Careful attention to system functionality and curated content facilitates opportunities to integrate AI-assisted chatbots into care delivery and public health education nationwide.