Abstract
Objective: To understand better the public perception and comprehension of medical technology such as artificial intelligence (AI) and robotic surgery. In addition to this, to identify sensitivity to their use to ensure acceptability and quality of counseling.
Subjects and Methods: A survey was conducted on a convenience sample of visitors to the MN Minnesota State Fair (n = 264). Participants were randomized to receive one of two similar surveys. In the first, a diagnosis was made by a physician and in the second by an AI application to compare confidence in human and computer-based diagnosis.
Results: The median age of participants was 45 (interquartile range 28–59), 58% were female (n = 154) vs 42% male (n = 110), 69% had completed at least a bachelor's degree, 88% were Caucasian (n = 233) vs 12% ethnic minorities (n = 31) and were from 12 states, mostly from the Upper Midwest. Participants had nearly equal trust in AI vs physician diagnoses. However, they were significantly more likely to trust an AI diagnosis of cancer over a doctor's diagnosis when responding to the version of the survey that suggested that an AI could make medical diagnoses (p = 9.32e-06). Though 55% of respondents (n = 145) reported that they were uncomfortable with automated robotic surgery, the majority of the individuals surveyed (88%) mistakenly believed that partially autonomous surgery was already happening. Almost all (94%, n = 249) stated that they would be willing to pay for a review of medical imaging by an AI if available.
Conclusion: Most participants express confidence in AI providing medical diagnoses, sometimes even over human physicians. Participants generally express concern with surgical AI, but they mistakenly believe that it is already being performed. As AI applications increase in medical practice, health care providers should be cognizant of the potential amount of misinformation and sensitivity that patients have to how such technology is represented.
Keywords: artificial intelligence, robotic surgery, active surveillance, patient counseling
Introduction
Recent advances in computer vision and machine learning have introduced a new wave of technologies to enhance the care of patients. Applications such as automatic measurement of features in medical imaging,1 precision medicine,2 and predictive methods describing the potential course of a disease3 are being actively developed. With such a wide variety of new applications there has been increased development in datasets and methods; however, relatively little research has been done on the public's perception of artificial intelligence (AI) in the care they receive. As a comparison, research into autonomous vehicles has been similarly rapid and much has been written about the public perception of autonomous vehicles.4,5 Small-scale initial research has been done on the intersection of medicine and AI from a consumer's perspective.6 This research found a negative association between AI and medical care in the minds of consumers. This seems to indicate a tension between the widespread research interest and the public interest in AI.
The advancement of AI has paralleled the development and introduction of robotic surgery. Laparoscopic surgery in urology has been on the forefront of new surgical technology with the use of robotics in procedures such as nephrectomy, prostatectomy, and cystectomy.7 Previous studies that examine patients' comfort and perception of such surgery8,9 provide a useful precedent when examining patients' view of AI. A common thread in this research has shown that there is misunderstanding about the capabilities and extent of current use, but there is a general optimism and prestige associated with new technologies. Boys et al. found that almost 20% of the respondents believed that the robot itself had some degree of autonomy, offering a natural topic in which to investigate individuals' perceptions of AI in medicine. The aim of this work is to explore public perceptions of AI in medicine, and evaluate relationships between demographic characteristics and disposition toward AI and robotics in the treatment and diagnosis of cancer.
Subjects and Methods
A questionnaire was developed to investigate the attitudes and acceptance of AI and robotic surgery in medicine. Since there is little research in this area, this survey was constructed to provide an initial estimate of individual attitudes and the distribution of beliefs that a clinician might encounter in the course of practice. This survey was designed with the oversight of experts in the clinical care of patients with urologic cancers and researchers in the field of computer science with expertise in the development of machine-learning algorithms. Topics were included on several application areas for AI in medicine, namely diagnostics, prognostics, and surgical robotics. The survey was administered to a convenience sample of attendees at the Minnesota State Fair on 2 days in 2019. Inclusion criteria were individuals 18 years or older who volunteered for and completed our questionnaire. Those who participated were given a nominal prize valued at <$5. The survey was self-administered digitally on tablets.
Demographic information was collected, including information on gender, age, level of highest education, race, ethnicity, occupation, and zip code. Additional demographic information relevant to the zip code as recorded in the 2017 American Community Survey10 was incorporated. These variables include the median household income, a designation of whether the zip code falls in a majority rural county, and the proportion of households that report having a broadband internet connection.
Participants were asked a series of questions related to the use of robotic surgery and AI applications in the treatment of cancer. The questions fall into four topics:
-
(1)
Active Surveillance
-
(2)
Autonomous Surgery
-
(3)
Conflicting Diagnosis Between a Doctor and AI
-
(4)
Consumer Facing Image Reading by AI
Each participant randomly received one of two versions of the survey. The only difference between the two versions was the first question, which (paraphrasing) was either (1) a radiologist or (2) an AI “believes there's a 25% chance your renal mass is cancer.” In both cases, the participant was recommended Active Surveillance. They were then asked whether they would pursue a second opinion. If “yes” they were given a follow-up question asking at what estimated confidence in benignity they would feel comfortable with active surveillance.
A second set of questions investigated the participants' perceptions of robotic surgery. First, they were asked to estimate what proportion of a robotic surgery is performed autonomously, and then they were asked to rate their comfort with robotic surgery on a Likert Scale.
The third section asked the participant to select the diagnosis in which they have the highest confidence between an AI and a doctor when the doctor believes a mass is cancerous whereas an AI predicts that it is benign. The last question asked whether the participant would be comfortable with using a web service that evaluated their medical imaging by using an AI algorithm to get a second opinion. If “yes” they were then asked what they would be comfortable paying for such a service.
Full survey text and answer options are described in Appendix A1.
Responses were captured and stored in a REDCap (REDCap; Vanderbilt University, Nashville, TN) database through the University of Minnesota and then exported for analysis in R version 3.4.4 (R Foundation for Statistical Computing, Vienna, Austria). Multivariate regressions were used as appropriate to identify associations between demographic factors and responses; p-values <0.05 were considered significant. This survey was administered with the approval and oversight of the Institutional Review Board of the University of Minnesota, and informed consent was obtained from all participants.
Results
Responses were collected from 298 participants over 2 days of data collection. After removing incomplete responses, a total of 264 participants were included in our study. Demographics of the participants are outlined in Table 1 for both survey versions 1 and 2. Survey response counts by question are found in Table 2.
Table 1.
Demographics
| Survey 1, n (%) | Survey 2, n (%) | Total, n (%) | |
|---|---|---|---|
| n | 128 | 136 | 264 |
| Gender | |||
| Female | 70 (55) | 84 (62) | 154 (58) |
| Male | 58 (45) | 52 (38) | 110 (42) |
| Education | |||
| High school diploma or GED | 6 (5) | 7 (5) | 13 (5) |
| Associate's degree | 9 (7) | 9 (7) | 18 (7) |
| Some college | 22 (17) | 25 (18) | 47 (18) |
| Bachelor's degree | 50 (39) | 53 (39) | 103 (39) |
| Graduate or professional degree | 41 (32) | 42 (31) | 83 (31) |
| Race | |||
| Asian | 9 (7) | 8 (6) | 17 (6) |
| Black or African American | 3 (2) | 2 (1) | 5 (2) |
| Hawaiian or Other Pacific Islander | 1 (1) | 0 (0) | 1 (0) |
| White | 109 (85) | 124 (91) | 233 (88) |
| Multiracial | 2 (2) | 1 (1) | 3 (1) |
| Other | 4 (3) | 1 (1) | 5 (2) |
| Zip code is majority rural | |||
| Not majority rural | 122 (95) | 127 (93) | 249 (94) |
| Majority rural | 4 (3) | 9 (7) | 13 (5) |
| Insufficient information | 2 (25) | 0 (0) | 2 (1) |
| Age | |||
| 18–30 | 37 (29) | 37 (27) | 74 (28) |
| 30–40 | 19 (15) | 23 (17) | 42 (16) |
| 40–50 | 21 (16) | 21 (14) | 42 (16) |
| 50–60 | 26 (20) | 19 (14) | 45 (17) |
| 60–70 | 16 (13) | 26 (19) | 42 (16) |
| 70–80 | 8 (6) | 9 (7) | 17 (6) |
| 80–90 | 1 (1) | 1 (1) | 2 (1) |
| Median household income of zip code (in thousands USD) | |||
| 0–25 | 1 (1) | 0 (0) | 1 (0) |
| 25–50 | 10 (8) | 21 (15) | 31 (12) |
| 50–75 | 55 (43) | 59 (43) | 114 (43) |
| 75–100 | 40 (31) | 37 (27) | 77 (29) |
| 100–125 | 19 (15) | 14 (10) | 33 (13) |
| 125–150 | 1 (1) | 2 (1) | 3 (1) |
| 150–175 | 0 (0) | 2 (1) | 2 (1) |
| Insufficient information | 2 (2) | 1 (1) | 3 (1) |
| Percent of households in zip code with broadband internet | |||
| 60–70 | 4 (3) | 3 (2) | 7 (3) |
| 70–80 | 20 (16) | 33 (24) | 53 (20) |
| 80–90 | 84 (66) | 81 (60) | 165 (63) |
| 90–100 | 18 (14) | 17 (13) | 35 (13) |
| Insufficient information | 2 (2) | 2 (1) | 4 (2) |
GED = General Educational Development exam.
Table 2.
Survey Responses (See Appendix A1 for Complete Question Text)
| Survey 1, n (%) | Survey 2, n (%) | Total, n (%) | |
|---|---|---|---|
| Active Surveillance Comfort Question 1 (Survey 1) | |||
| “Yes, I would seek another opinion” | 72 (56) | — | |
| “No, I would follow the recommendation to watch the mass” | 56 (44) | — | |
| Active Surveillance Question 1 (Survey 2) | |||
| “Yes, I would seek another opinion” | — | 91 (67) | |
| “No, I would follow the recommendation to watch the mass” | — | 45 (33) | |
| Active Surveillance Question 2 | |||
| “76–80%” | 15 (12) | 21 (15) | 36 (14) |
| “81–85%” | 4 (6) | 6 (4) | 10 (4) |
| “86–90%” | 7 (5) | 4 (3) | 11 (4) |
| “91–95%” | 10 (8) | 19 (14) | 29 (11) |
| “96–99%” | 21 (16) | 21 (15) | 42 (16) |
| “>99%” | 15 (12) | 20 (15) | 35 (13) |
| Not asked | 56 (44) | 45 (33) | 101 (38) |
| Autonomous Surgery Question 1 | |||
| Mean: | 35.3 | 41.2 | 38.4 |
| 25th percentile | 14.0 | 22.8 | 18.0 |
| 50th percentile | 30.0 | 36.0 | 31.5 |
| 75th percentile | 53.0 | 66.8 | 66.0 |
| Autonomous Surgery Question 2 | |||
| “Completely uncomfortable” | 20 (16) | 27 (20) | 47 (18) |
| “Somewhat uncomfortable” | 50 (39) | 48 (35) | 98 (37) |
| “Neither comfortable or uncomfortable” | 9 (7) | 16 (12) | 25 (9) |
| “Somewhat comfortable” | 38 (30) | 30 (22) | 68 (26) |
| “Completely comfortable” | 11 (9) | 15 (11) | 26 (10) |
| AI vs Physician Question 1 | |||
| “The doctor” | 75 (59) | 45 (33) | 120 (45) |
| “The artificial intelligence/computer algorithm” | 53 (41) | 91 (67) | 144 (55) |
| Second Opinion AI Question 1 | |||
| “Yes” | 120 (94) | 129 (95) | 249 (94) |
| “No” | 8 (6) | 7 (5) | 15 (6) |
| Second Opinion AI Question 2 | |||
| “$0” | 11 (9) | 9 (7) | 20 (8) |
| “$1–10” | 5 (4) | 6 (4) | 11 (4) |
| “$11–20” | 10 (8) | 11 (8) | 21 (8) |
| “$21–50” | 22 (17) | 24 (18) | 46 (17) |
| “$51–100” | 39 (30) | 43 (32) | 82 (31) |
| “>$100” | 33 (26) | 36 (26) | 69 (26) |
| Not asked | 8 (6) | 7 (5) | 15 (6) |
AI = artificial intelligence.
Active surveillance
Of the 264 responses, 163 individuals (62%) indicated that they would seek a second opinion given the recommendation to wait on a surgery and watch the growth of the mass via imaging and the rest indicated that they would follow the recommendation. Of those who received the version of the question indicating that the recommendation was based on the evaluation of an AI algorithm, 67% indicated that they would seek a second opinion compared with 56% when the recommendation was based on a physician's recommendation. This was not found to be significant in a chi-square test (p = 0.098); however, a future study may find significant differences in this question. In multivariate analysis, no demographic features were found to be significant in predicting whether the participant would seek a second opinion.
Of those who indicated that they would seek a second opinion, a plurality of 42 out of 101 (42%) indicated that they would prefer a confidence level between 96% and 99%. See Table 2 for a complete summary of responses. Lower levels of education were correlated with lower levels of confidence required: (in order increasing absolute value of coefficients) those with some college education (p = 0.032), those with an associate's degree (p = 0.024), and those with a high school or GED (p = 0.0074) (Table 3).
Table 3.
Multivariate Regression of Percentage Confidence Required to Feel Comfortable with Active Surveillance
| Regression coefficient | p | |
|---|---|---|
| Intercept | 0.94 | < 0.001 |
| Age | −9.3e-05 | 0.82 |
| Survey | ||
| Received Survey 2 | −0.0025 | 0.86 |
| Gender | ||
| Male | 0.0015 | 0.92 |
| Education | ||
| High school diploma or GED | −0.084 | 0.0074 |
| Associate's degree | −0.064 | 0.024 |
| Some college | −0.042 | 0.032 |
| Graduate or professional degree | −0.031 | 0.062 |
| Race | ||
| Not white | −0.0084 | 0.71 |
| Rural/urban | ||
| Zip code is majority rural | −0.0095 | 0.76 |
| Zip code median household income (in thousands) | 6.2e-04 | 0.19 |
| Percent of households in zip code with broadband internet | −6.4e-04 | 0.73 |
Boldface values indicate statistical significance.
Autonomous surgery
Despite the fact that there are no U.S. Food and Drug Administration (FDA) approved autonomous robotic surgical systems in the United States, when asked to estimate the percentage of a standard robotic surgery that was autonomous, participants selected values from 0% to 100%. The median value selected was 31.5%, and the mean value was 38.4%. No demographics were found to be significant in multiple regression against this estimation.
With regard to comfort with robotic surgery, 47 participants (18%) indicated that they were “completely uncomfortable” with robotic surgery, 98 (37%) “somewhat uncomfortable,” 25 (9%) “neither comfortable or uncomfortable,” 68 (26%) “somewhat comfortable,” and 26 (10%) “completely comfortable.” Results of multiple regression can be found in Table 4. Older respondents and those whose zip codes had a higher percentage of households with a broadband internet subscription were significantly more likely to choose “completely uncomfortable” vs other responses. The odds ratio (OR) of age was 1.03 (p = 0.0061), and the percentage of households with broadband internet had OR 1.13 (p = 0.032). Those who indicated zip codes with higher median incomes were much less likely to select “completely uncomfortable,” with OR 0.57 (p = 0.00078). Higher median incomes were more likely to select “Somewhat comfortable” vs other options (OR = 1.03, p = 0.014). Being male (OR = 5.02, p = 0.0010) and having an associate's degree (OR = 4.36, p = 0.037) were significant among those selecting “Completely comfortable” vs all other options.
Table 4.
Multivariate Logistic Regression of Each Level of Comfort with Robotic Surgery
| Odds ratio | p | |
|---|---|---|
| Completely uncomfortable | ||
| (Intercept) | <0.01 | 0.025 |
| Age | 1.03 | 0.0061 |
| Survey | ||
| Received Survey 2 | 1.32 | 0.43 |
| Gender | ||
| Male | 0.56 | 0.11 |
| Education | ||
| High school diploma or GED | 0.26 | 0.24 |
| Associate's degree | 0.60 | 0.47 |
| Some college | 0.57 | 0.30 |
| Graduate or professional degree | 0.89 | 0.77 |
| Race | ||
| Not white | 1.93 | 0.24 |
| Rural/urban | ||
| Zip code is majority rural | <0.01 | 0.99 |
| Zip code median household income (in thousands) | 0.57 | 0.00078 |
| Percent of households in zip code with broadband internet | 1.13 | 0.032 |
| Somewhat uncomfortable | ||
| Intercept | 1.05 | 0.99 |
| Age | 1.01 | 0.067 |
| Survey | ||
| Received Survey 2 | 0.89 | 0.67 |
| Gender | ||
| Male | 0.77 | 0.35 |
| Education | ||
| High school diploma or GED | 0.81 | 0.74 |
| Associate's degree | 0.45 | 0.19 |
| Some college | 0.96 | 0.91 |
| Graduate or professional degree | 0.81 | 0.52 |
| Race | ||
| Not white | 1.68 | 0.21 |
| Rural/urban | ||
| Zip code is majority rural | 0.71 | 0.60 |
| Zip code median household income (in thousands) | 1.00 | 0.72 |
| Percent of households in zip code with broadband internet | 1.01 | 0.85 |
| Somewhat comfortable | ||
| Intercept | 1.41 | 0.92 |
| Age | 1.01 | 0.47 |
| Survey | ||
| Received Survey 2 | 0.61 | 0.099 |
| Gender | ||
| Male | 1.23 | 0.50 |
| Education | ||
| High school diploma or GED | 2.64 | 0.13 |
| Associate's degree | 0.98 | 0.97 |
| Some college | 1.02 | 0.97 |
| Graduate or professional degree | 1.32 | 0.44 |
| Race | ||
| Not white | 0.58 | 0.30 |
| Rural/urban | ||
| Zip code is majority rural | 2.06 | 0.26 |
| Zip code median household income (in thousands) | 1.03 | 0.014 |
| Percent of households in zip code with broadband internet | 0.95 | 0.32 |
| Completely comfortable | ||
| Intercept | 3.50 | 0.79 |
| Age | 1.00 | 0.95 |
| Survey | ||
| Received Survey 2 | 1.25 | 0.63 |
| Gender | ||
| Male | 5.02 | 0.0010 |
| Education | ||
| High school diploma or GED | 1.41 | 0.77 |
| Associate's degree | 4.36 | 0.037 |
| Some college | 0.88 | 0.85 |
| Graduate or professional degree | 0.79 | 0.69 |
| Race | ||
| Not white | <0.01 | 0.99 |
| Rural/urban | ||
| Zip code is majority rural | 2.98 | 0.18 |
| Zip code median household income (in thousands) | 1.00 | 0.94 |
| Percent of households in zip code with broadband internet | 0.95 | 0.46 |
Boldface values indicate statistical significance.
Taken together as a continuous measure of comfort (ranging from 1 to 5) being male (p = 0.0017), having an associate's degree (p = 0.041), and indicating a zip code with a higher median income (p = 0.0012) all again had a positive relationship with comfort with robotic surgery. There was a negative association between selecting a race other than white (p = 0.014) and neighborhoods with a higher number of households with a broadband internet subscription (p = 0.035).
AI vs physician
Given the scenario that a physician and an AI algorithm disagree in their estimation that a mass is benign (the doctor believing it to be cancerous and the AI believing it to be benign), roughly half of respondents, 144 out of 264, indicate a higher confidence in the AI. Those who randomly received the second version of the survey, where an earlier question suggested that AI can provide recommendations, were much more likely to trust the AI's diagnosis over the doctor's (OR = 3.45, p ≤ 0.001). The majority of participants who received the first survey had greater confidence in the doctor's diagnosis whereas the reverse was true for those who received the second, as highlighted in Figure 1. No other predictors were found to be significant. Complete multiple regression results can be found in Table 5.
FIG. 1.
Percent of participants in each survey that selected greater confidence in either the doctor's diagnosis or an AI diagnosis given the doctor (see Appendix A1 for full question text). AI = artificial intelligence.
Table 5.
Multivariate Logistic Regression of Preference for a Physician's Recommendation Over an Artificial Intelligence
| Odds ratio | p | |
|---|---|---|
| Intercept | 1090 | 0.022 |
| Age | 1.00 | 0.81 |
| Survey | ||
| Received survey 2 | 0.29 | <0.001 |
| Gender | ||
| Male | 1.18 | 0.55 |
| Education | ||
| High school diploma or GED | 2.38 | 0.20 |
| Associate's degree | 0.61 | 0.37 |
| Some college | 1.22 | 0.60 |
| Graduate or professional degree | 0.64 | 0.18 |
| Race | ||
| Not white | 0.81 | 0.63 |
| Urban/rural | ||
| Zip code is majority rural | 2.20 | 0.24 |
| Zip code median household income (in thousands) | 1.00 | 0.69 |
| Percent of households in zip code with broadband internet | 0.92 | 0.054 |
Boldface values indicate statistical significance.
Second opinion AI
Almost all participants (94%) indicated that they would be comfortable with sending their medical imaging to an AI algorithm for review via a website. Of those individuals, 92% indicated that they would be willing to pay for such a service. From lowest to highest: 11 (4%) selected “$1–10,” 21 (8%) selected “$11–20,” 46 (18%) selected “$21–50,” 82 (33%) selected “$51–100,” and 69 (28%) indicated that they would be willing to pay more than $100. No significant relationships were found between demographic variables and individuals' responses in this category.
Discussion
The results of this survey show a great diversity in opinions on AI and robotic surgery in the diagnosis and treatment of cancer. Lower levels of education were correlated with lower certainty required in the benign nature of a mass to be comfortable with waiting on surgery. A number of factors were significantly associated with comfort with robotic surgery, including lower age, Caucasian race, lack of broadband internet connections, higher median income areas, having only an associate's degree, and male gender. Being young and male seems to align with a narrative of higher tolerance to risk taking; however, other significant relationships seem to be more nuanced. It is plausible that those in higher median income neighborhoods may have higher exposure to consumer robotics, which may translate to comfort with robotic surgery. The other relationships are more difficult to reason about, and further research would be merited.
The most notable relationship found in this survey was the positive association between receiving the version of the survey that posited an AI estimate in the first question and having higher confidence in the diagnosis of an AI when it disagreed with the physician.
In spite of the extensive adoption of robotic surgery in the United States, there is still significant misunderstanding about its nature. Even though surgical robots do not currently operate autonomously in standard practice, 88% of participants believe that it does, at least to some degree. Speculatively, this may relate to media portrayal of robots as having AI. Particular to robotic surgery, many informational videos display the movements of the robot without the corresponding movements of the surgeon in frame. This may cause a patient to attribute actions to the robot rather than the surgeon.
In contrast with initial customer surveys,6 we found evidence to suggest a certain openness to AI applications. This finding aligns with recent research suggesting that public perception and optimism of AI as a whole has risen markedly.11 A third of all participants believed that robotic surgery contained some autonomous behavior and indicated that they were either “Somewhat comfortable” or “Completely comfortable” with such a procedure. Almost a third of individuals who received a recommendation to wait and monitor a tumor's growth based on an AI estimation were comfortable with following the recommendation. Many patients could feel that an AI gives them additional certainty in their diagnosis, which has been shown to be predictive of the quality of life in individuals undergoing monitoring of small renal tumors.12
This survey has limitations. The sample available for this study reflects a regional bias to the upper Midwest and its corresponding demographic characteristics. Those who took the survey may not correspond with the population of patients seeking care for cancer. The significant impact of the first question may indicate a secondary bias that may be elucidated with further study. As an initial investigation into attitudes surrounding AI and medical applications, this survey touches briefly on a wide variety of topics. A more focused, validated survey to measure sensitivity to information about medical AI and patient counseling would be a good direction for future research.
The significant priming effect of the first question suggests a sensitivity to how an AI diagnosis is presented. Those who receive an indication that an AI can make a diagnosis show greater confidence in it. It is possible that the reverse, a negative priming effect, could also be observed. Given the widespread research, there has been significant media coverage. In May 2019, the New York Times published an article headlined “A.I. Took a Test to Detect Lung Cancer. It Got an A.”13 Recently, Google AI announced a partnership with the Mayo Clinic.14 Such media coverage has the possibility of increasing the public's confidence and inflates the capabilities of such technology.
Given public optimism about AI and the significant confusion about current technological capabilities, there is a great need for careful patient counseling. Patients may feel overly confident during a course of treatment given the widespread news of success in AI research. This may lead them to take risks that they otherwise would not feel comfortable with. It is, therefore, important to discuss with patients the role that technology is playing in their care, along with its limitations.
Conclusion
There is significant variance in population perception and understanding of AI and robotic surgery. Proactive conversations with patients who might be undergoing robotic surgery may be beneficial. Our results suggest that individuals' opinions and confidence in AI are very sensitive to how AI is described. As research continues in AI applications, there is a need for clear communication going forward about the strengths and limitations of such techniques.
Acknowledgment
Additional support was provided by the Climb for Kidney Cancer Foundation.
Abbreviations Used
- AI
artificial intelligence
- CT
computed tomography
- FDA
U.S. Food and Drug Administration
- GED
General Educational Development exam
- OR
odds ratio
Appendix
Appendix A1. Survey Text
Survey 1
Questions and responses for Survey 1 are as follows.
Questions on active surveillance
“Suppose you begin to experience stomach pain and decide to visit a medical clinic. Your doctor orders an abdominal CT scan which shows a 3 cm (1.5 inch) mass that could be a kidney cancer (see image below). She then refers you to a urologist for a consult for surgical removal of the mass, which would have a 99% chance of cure if it is cancer. The urologist explains that masses with CT characteristics like yours are unlikely to be cancerous, estimating the chance this mass is a kidney cancer is only 25%. He suggests that you undergo follow-up imaging every six months moving forward, and that if the mass changes considerably he will surgically remove it. Would you seek a second opinion?”
“Yes, I would seek another opinion”
“No, I would follow the recommendation to watch the mass”
“If you feel uncomfortable not treating a mass with a 25% chance of cancer, what level of certainty do you need to have before your feel comfortable not treating a mass?”
“76–80%,” “81–85%,” “86–90%,” “91–95%,” “96–99%,” “>99”
Autonomous surgery
“Today, many surgical procedures are done with the assistance of a robot. In the average robotic assisted surgery, what proportion of the procedure do you estimate that the robot is operating autonomously (or in other words, by itself without guidance from the surgeon)?”
(User input slider 0–100)
“If a robot were developed to perform a cancer surgery completely autonomously (in other words, the robot performs the surgery by itself without guidance from a surgeon). What would your comfort level be to undergo that type of operation?”
“Completely uncomfortable,” “Somewhat uncomfortable,” “Neither comfortable or uncomfortable,” “Somewhat comfortable,” “Completely comfortable”
AI vs physician
“Suppose you visit your clinic for nausea and your doctor orders a CT scan of the abdomen. Your doctor then reviews your imaging and doesn't find an explanation for your nausea, but unexpectedly finds an adrenal mass which she believes is cancer. However, an artificial intelligence computer algorithm also reviews your imaging and concludes your adrenal mass is benign. In which diagnosis do you have a higher confidence?”
“The doctor,” “The artificial intelligence/computer algorithm”
Second opinion AI
“If there was a website where an artificial intelligence/computer algorithm was able to read your CT scan to give you a second opinion on whether a mass is cancer or not, would you allow your CT scan to be read by a computer?”
“Yes,” “No”
“If you would allow a computer to read your CT scan how much would you be willing to pay for that service?”
“$0,” “$1–10,” “$11–20,” “$21–50,” “$51–100,” “>$100”
Survey 2
Questions on active surveillance
“Suppose you begin to experience stomach pain and decide to visit a medical clinic. Your doctor orders an abdominal CT scan which shows a 3 cm (1.5 inch) mass that could be a kidney cancer (see image below). She then refers you to a urologist for a consult for surgical removal of the mass, which would have a 99% chance of cure if it is cancer. The urologist explains that an artificial intelligence/computer algorithm concluded that masses with CT characteristics like yours are unlikely to be cancerous, estimating the chance this mass is a kidney cancer is only 25%. He suggests that you undergo follow-up imaging every six months moving forward, and that if the mass changes considerably he will surgically remove it. Would you seek a second opinion?”
“Yes, I would seek another opinion,”
“No, I would follow the recommendation to watch the mass”
“If you feel uncomfortable not treating a mass with a 25% chance of cancer, what level of certainty do you need to have before your feel comfortable not treating a mass?”
“76–80%,” “81–85%,” “86–90%,” “91–95%,” “96–99%,” “>99”
Autonomous surgery
“Today, many surgical procedures are done with the assistance of a robot. In the average robotic assisted surgery, what proportion of the procedure do you estimate that the robot is operating autonomously (or in other words, by itself without guidance from the surgeon)?”
(User input slider 0–100)
“If a robot were developed to perform a cancer surgery completely autonomously (in other words, the robot performs the surgery by itself without guidance from a surgeon). What would your comfort level be to undergo that type of operation?”
“Completely uncomfortable,” “Somewhat uncomfortable,” “Neither comfortable or uncomfortable,” “Somewhat comfortable,” “Completely comfortable”
AI vs physician
“Suppose you visit your clinic for nausea and your doctor orders a CT scan of the abdomen. Your doctor then reviews your imaging and doesn't find an explanation for your nausea, but unexpectedly finds an adrenal mass which she believes is cancer. However, an artificial intelligence computer algorithm also reviews your imaging and concludes your adrenal mass is benign. In which diagnosis do you have a higher confidence?”
“The doctor,” “The artificial intelligence/computer algorithm”
Second opinion AI
“If there was a website where an artificial intelligence/computer algorithm was able to read your CT scan to give you a second opinion on whether a mass is cancer or not, would you allow your CT scan to be read by a computer?”
“Yes,” “No”
“If you would allow a computer to read your CT scan how much would you be willing to pay for that service?”
“$0,” “$1–10,” “$11–20,” “$21–50,” “$51–100,” “>$100”
Disclaimer
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Author Disclosure Statement
No competing financial interests exist.
Funding Information
Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA225435.
References
- 1. Thrall JH, Li X, Li Q, et al. . Artificial intelligence and machine learning in radiology: Opportunities, challenges, pitfalls, and criteria for success. J Am Coll Radiol 2018;15:504–508 [DOI] [PubMed] [Google Scholar]
- 2. Krittanawong C, Zhang H, Wang Z, et al. . Artificial intelligence in precision cardiovascular medicine. J Am Coll Cardiol 2017; 69:2657–2664 [DOI] [PubMed] [Google Scholar]
- 3. Cestari A. Predictive models in urology. Urologia 2013; 80:42–45 [DOI] [PubMed] [Google Scholar]
- 4. Choi JK and Ji YG. Investigating the importance of trust on adopting an autonomous vehicle. Int J Human–Comp Interact 2015;31:692–702 [Google Scholar]
- 5. Cavoli C, Phillips B, Cohen T et al. . Social and Behavioural Questions Associated with Automated Vehicles: A Literature Review. London: UCL Transport Institute, 2017. [Google Scholar]
- 6. Longoni C, Bonezzi A, and Morewedge CK. Resistance to medical artificial intelligence. 2019. https://academic.oup.com/jcr/advance-article/doi/10.1093/jcr/ucz013/5485292
- 7. Rassweiler JJ, Autorino R, Klein J, et al. . Future of robotic surgery in urology. BJU Int 2017; 120:822–841 [DOI] [PubMed] [Google Scholar]
- 8. Ahmad A, Ahmad ZF, Carleton JD, et al. . Robotic surgery: Current perceptions and the clinical evidence. Surg Endosc 2017;31:255–263 [DOI] [PubMed] [Google Scholar]
- 9. Boys JA, Alicuben ET, DeMeester MJ, et al. . Public perceptions on robotic surgery, hospitals with robots, and surgeons that use them. Surg Endosc 2016;30:1310–1316 [DOI] [PubMed] [Google Scholar]
- 10. American Community Survey Office. 2017. data profiles. 2017. https://www.census.gov/acs/www/data/data-tables-and-tools/data-profiles/
- 11. Fast E and Horvitz E.. Long-term trends in the public perception of artificial intelligence. http://arxiv.org/abs/1609.04904
- 12. Parker PA, Alba F, Fellman B, et al. . Illness uncertainty and quality of life of patients with small renal tumors undergoing watchful waiting: A 2-year prospective study. Eur Urol 2013;63 :1122–1127 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Grady D. A.I. took a test to detect lung cancer. It got an A. ISSN 0362-4331. https://www.nytimes.com/2019/05/20/health/cancer-artificial-intelligence-ct-scans.html
- 14. Mayo clinic selects google as strategic partner for health care innovation, cloud computing. https://newsnetwork.mayoclinic.org/discussion/mayo-clinic-selects-google-as-strategic-partner-for-health-care-innovation-cloud-computing/