“Alexa, Am I pregnant?”: A content analysis of a virtual assistant’s responses to prenatal health questions during the COVID-19 pandemic

Abstract

Objective

To elucidate whether Amazon’s virtual assistant, Alexa, provides evidence-based support as a supplement to provider-facilitated prenatal care, during the COVID-19 pandemic.

Methods

Using a conceptual content analysis approach, a query of 40 questions, relating to all phases of pregnancy, was collected from Alexa by two independent investigators, using two unique devices, over a one-week period between May 20, 2020 and May 27, 2020. Alexa’s responses were matched to the evidence-based content from the American College of Obstetricians and Gynecologists (ACOG) and reviewed by a Certified Nurse Midwife for completeness and currency.

Results

Of the 40 questions asked of Alexa, it was unable to answer 14 questions (35%). A total of 21 out of the 40 responses (52%) were not evidence-based and three COVID-specific questions (about 1%) were answered incorrectly or insufficiently. Four questions (10%) were answered accurately.

Conclusion

Alexa was largely unable to provide evidence-based answers to commonly asked pregnancy questions and, in many cases, supplied inaccurate, incomplete, or completely unrelated answers that could further confuse health consumers.

Practice Implications

Ensuring that mobile health (mhealth) tools, such as Amazon Alexa, are evidence-based and credible in answering common prenatal questions has important implications for this pandemic and future consumer needs.

1. Introduction

Frequent prenatal visits are a hallmark of pregnancy care. The Office on Women’s Health suggests that pregnant individuals attend approximately 14 prenatal appointments during a 40-week pregnancy [1]. The Novel Coronavirus SARS CoV-2 (COVID-19) pandemic transformed prenatal care in the United States in an effort to decrease patient exposure. A modified format for prenatal visits was documented at a COVID-19 epicenter, New York City. These guidelines suggest one, in-person, prenatal visit at weeks’ 11–13, 18–22, 27–28, 36, 39, and 40, with an additional seven to eight telehealth visits, spaced throughout the pregnancy [2]. An alternative paradigm, from the University of Michigan, suggests a total of four in-person visits, one ultrasound, and four virtual visits [3]. Often, pregnant patients are asked to purchase a blood pressure cuff for home monitoring [4]. In fact, similar models of a reduced prenatal care schedule, or a schedule with virtual, phone, and online support, have been shown to have comparable quality of care, versus traditional care, in a recent randomized controlled trial at the Mayo Clinic [5,6].

Women are increasingly utilizing mobile health (mhealth) for prenatal health information. The widespread use of smartphones [7] has led to a research-based call for the inclusion of such technologies [8]. These mhealth tools may be a suitable adjunct to formal care if they provide high quality, evidence-based information such as vitamin consumption, nutrition tips, and sleep information. A study of 193 women seeking prenatal care found that the majority of respondents utilized apps for their convenience and accessibility [9]. First-time mothers were the most frequent health app users, however, users expressed concern over the credibility of the apps [9]. A meta-analysis of 15 randomized controlled trials, related to prenatal information on mobile apps, found that these technologies had the potential to promote maternal well-being in a variety of areas, including preparedness and knowledge, but the credibility of included content could not be confirmed [10].

Virtual assistants, like Amazon Alexa and Google Home, exist in the homes of approximately 50 million Americans [11]. A survey of over 1000 virtual assistant users found that the majority of respondents used their virtual assistant multiple times per day [11]. Although not specifically related to health, almost two-thirds of users used the device to seek information and females made up a higher percentage of users [11]. It is critical to understand the role and reliability of mhealth in prenatal care, especially with the recent changes to health care delivery. The goal of this research is to elucidate whether Amazon’s virtual assistant, Alexa, provides evidence-based information in response to common pregnancy questions.

2. Methods

A list of frequently asked prenatal questions were curated from the American College of Obstetricians and Gynecologists’ (ACOG) Frequently Asked Questions webpage [12] and reviewed for relevance by a Certified Nurse Midwife (CNM). Three COVID-specific questions were added to address the current health crisis. Forty questions were queried of Alexa including: confirming a pregnancy, identifying warning signs, and recognizing the signs of labor (see Appendix). The investigators, using two unique devices, independently asked Alexa the same set of questions and recorded the responses. In addition to the responses, unsolicited data sources provided by Alexa were recorded. The query took place during the COVID-19 pandemic, over a one-week period between May 20, 2020 and May 27, 2020. Alexa’s responses were transcribed verbatim and their accuracy confirmed using the associated Alexa app, which transcribes the responses and sources. Alexa was not provided prompts or contextual information. Alexa’s responses were compared to ACOG’s prenatal guidelines to verify accuracy and completeness by two independent coders, including a CNM, a content expert [12].

3. Results

Conceptual content analysis was conducted employing explicit coding [13] and dichotomous categorization of all 40 responses for accuracy (accurate or inaccurate) and completeness (complete or incomplete). As both coders compared Alexa’s responses with ACOG’s evidence-based information, there was complete agreement between the coders. Alexa supplied identical responses to 36 out of the 40 questions (90%) (see Appendix). Overall, four responses (10%) were accurate relating to: changing cat litter, caffeine consumption, preeclampsia, and foods to avoid. Alexa was unable to answer 14 questions (35%) responding, “Sorry, I don’t know that one” or “Hmm.. I don’t know that.” The unanswered questions included, “How do I know if I’m in labor?” and “Is it safe to smoke while pregnant?”

A total of 21 out of 40 responses (52%) were not accurate based on current evidence. An additional response was deemed incomplete, only listing some of the relevant nutrients needed in prenatal vitamins (see Fig. 1 ). Inaccurate answers were tangentially related to the question, such as when asked whether it is safe to drink alcohol during pregnancy, Alexa replied, “Non-alcoholic beer has not been proven safe for consumption during pregnancy.” For six questions (15%) in the inaccurate category, the responses were not relevant to the question. For example, when asked about the safety hot tub use, Alexa replied, “You can get pregnant when having sex in a hot tub, bath tub, or hot springs with high water temperature.” When asked which vaccines were safe during pregnancy, Alexa replied, “You can't get the vaccine while you're pregnant, and contracting the measles during your pregnancy can be very dangerous for your baby.”

Fig. 1.

Example Alexa responses to common prenatal queries.

The three COVID-specific questions were answered incorrectly or insufficiently. When Alexa was asked about the main symptoms of COVID-19 in pregnancy, it reported information on cholestasis, “intense itching without a rash is the main symptom of intrahepatic cholestasis…” When asked, “Should I go to my OB appointment during the COVID-19 pandemic?” and “Should I give birth outside of a hospital during the COVID-19 pandemic?,” Alexa replied with information about when and where COVID-19 was identified, how it spreads and information on donating to those affected. Alexa’s answers to one COVID-19 question varied between the researchers. For one researcher, Alexa replied to the question about seeking in-person OB care during the pandemic, “Sorry, I don’t know that one,” but for the other researcher, it provided an identical answer to each of the aforementioned COVID-19 questions (see Appendix).

Alexa often noted its source in the response (i.e., “According to the Mayo Clinic” or “Here is something I found on Reference.com”), if a reference was not provided in the response, the Alexa app supplied the sources. The most frequently reported source was Reference.com (n = 9), which is not a reliable source of maternal health information. Alexa also referred to Wikipedia or Wikihow (n = 4), VeryWellFamily (n = 3), Answers.com (n = 1), BabyCenter (n = 1), or BabyMed (n = 1). On four occasions, the app offered, “This information is not medical advice. Consult a healthcare professional if you have a medical problem.” This response was supplied to questions related to vaginal bleeding, heartburn, symptoms of preeclampsia, and regularity of fetal movement (see Appendix).

4. Discussion and conclusion

4.1. Discussion

Women are increasingly using mhealth and virtual tools for information [14]. Recent studies indicate a sense of empowerment and control over one’s health through the use of mhealth technology [8]. Text messaging programs, such as text4baby, were designed to provide women with accurate prenatal health information in a readily available mhealth platform [15]. One randomized controlled trial of text4baby found it helpful in reducing alcohol consumption during pregnancy [15].

Overall, Alexa was unable to provide answers to commonly asked prenatal questions and often supplied inaccurate, incomplete, or completely unrelated answers that could further confuse health consumers. Many of these answers can readily be found on evidence-based websites (i.e. ACOG and womenshealth.gov), but very few evidence-based sources were consulted by Alexa. In contrast, other mhealth tools, like the “What to Expect” mobile app includes answers to questions curated by health professionals.

In this digital era, when many people rely on virtual assistants to supply basic, yet specific, information, it is surprising that Alexa does not have the ability to search reputable sites for information related to pregnancy. It is important for health care providers to point out mhealth limitations and offer reliable sources for information. Further, healthcare providers must advocate for virtual platforms to improve the accuracy and reliability of the information provided.

4.2. Conclusion

The pandemic related disruption in prenatal care demonstrates the opportunity to utilize mhealth to support patients. Public reliance on technology and mhealth tools is well established. Ensuring these tools are credible for answering frequently asked prenatal questions has important implications for this pandemic and future patient care needs.

4.3. Practice implications

Consumer use of inaccurate virtual assistants to obtain health care information can create false knowledge and subsequent anxiety about what they have learned and how to proceed. On the contrary, an astute patient consumer, using reliable sources, can productively advocate for themselves and their family. It is imperative that media sources provide evidence-based information and that health care providers inform their patients of accurate, reliable sources [16]. Virtual assistants that render evidence-based information can improve the health outcomes. Future research on the reliability of information offered by virtual assistants is warranted. Likewise, the exploration of consumer confidence of and reliance on health information from virtual assistants, like Alexa, is needed.

Author contribution statement

J. Schindler-Ruwisch: conceptualization, data curation, formal analysis, methodology, writing - original draft.

C. Palancia Esposito: data curation, formal analysis, validation, writing - review & editing.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of Competing Interest

The authors report no declarations of interest.

Appendix A. Complete list of Alexa queries (N = 40)

Pregnancy Question
1. How do I know if I'm pregnant?
2. When should I start taking prenatal vitamins?a
3. What should be in prenatal vitamins?
4. What should I do if my home pregnancy test is positive?
5. When should I seek prenatal care?
6. Where should I seek prenatal care?
7. Is it safe to drink alcohol in pregnancy?
8. How much alcohol is safe to drink in pregnancy?
9. How much caffeine can I have while pregnant?b
10. What medicines can I take for pain while pregnant?
11. Is it safe to have sex while pregnant?
12. Is it safe to exercise while pregnant?
13. Is bleeding during pregnancy normal?
14. How do I relieve constipation during pregnancy?
15. How do I relieve heartburn during pregnancy?c
16. When will morning sickness go away?
17. I'm pregnant and vomiting, what should I do?
18. I'm pregnant and dizzy, what should I do?
19. When should I feel the baby start kicking in pregnancy?
20. How often should I feel the baby move in pregnancy?
21. What foods should I avoid while pregnant?
22. What is the best position to sleep in while pregnant?
23. Is it safe to go in a hot tub while pregnant?
24. What activities should I avoid while pregnant?
25. Is it safe to smoke while pregnant?
26. Is it safe to dye my hair while pregnant?
27. Is it safe to paint while pregnant?
28. Is it safe to change cat litter while pregnant?
29. What vaccines should I get while pregnant?
30. Can I fly on a plane in my third trimester?
31. When should I get an ultrasound in pregnancy?
32. How do I know if I'm in labor?
33. How do I know if my water broke?
34. How far apart should my contractions be when I go to the hospital?
35. What are signs of anemia in pregnancy?
36. What are signs of preeclampsia?
37. Is it normal to have swollen feet during your third trimester?
38. What are symptoms of COVID-19 in pregnancy?
39. Should I go to my OB appointment during the COVID-19 pandemic?
40. Should I give birth outside of a hospital during the COVID-19 pandemic?d

^aAlexa responded to this query with a musical album for one researcher. For the second researcher, a more relevant response was provided, “Here's something I found on Reference.cm: the best time to take vitamins is in the morning, according to Mother Nature Network. Upon waking from sleep, the body needs nutrients for energy to start the activities of the day. Vitamin supplements are synthetic substances that break down and are absorbed into the body best with food.” The latter, more relevant answer, was coded.

^bAlexa responded to one researcher with, “According to First Databank: Using caffeine while pregnant may have risks. For medical advice, always consult your healthcare professional.” Alexa responded to the second researcher with, “Here is something I found from the article "Caffeine" on Wikipedia: The UK Food Standards Agency has recommended that pregnant women should limit their caffeine intake, out of prudence, to less than 200 mg of caffeine a day the equivalent of two cups of instant coffee, or one and a half to two cups of fresh coffee.” The latter, more detailed reply, was coded.

^cFor one researcher Alexa replied, “Here's something I found on howtostuff.com: In a similar manner; reflexology can speed up labor, but it's also helpful for relieving pregnancy symptoms like heartburn. This information is not medical advice. Consult a healthcare professional if you have a medical problem.” For the other researcher, Alexa replied, “Here's something I found on Wikihow: Step 1-Avoid foods that commonly trigger heartburn, Step 2-Eat smaller meals, Step 3-Don't eat too close to bedtime, Step 4-Avoid beverages that cause gastrointestinal distress.” The latter, more relevant reply, was coded.

^dFor one researcher, Alexa replied, “Sorry, I don’t know that one” and for the other researcher, an identical response to the previous COVID question (question #39) was supplied. The latter, complete answer, was coded.