Accuracy of a Popular Online Symptom Checker for Ophthalmic Diagnoses

Key Points

Question

What is the accuracy of a popular online symptom checker for ophthalmic diagnoses?

Findings

In this vignette-based, cross-sectional study, the top 3 diagnoses generated by the online symptom checker included the correct diagnoses in 16 of 42 (38%) cases. A substantial number of diagnoses may not be captured.

Meaning

This finding suggests that patients should exercise caution if depending on online symptom checkers to self-diagnosis ophthalmic conditions.

Abstract

Importance

Because more patients are presenting with self-guided research of symptoms, it is important to assess the capabilities and limitations of these available health information tools.

Objective

To determine the accuracy of the most popular online symptom checker for ophthalmic diagnoses.

Design, Setting, and Participants

In a cross-sectional study, 42 validated clinical vignettes of ophthalmic symptoms were generated and distilled to their core presenting symptoms. Cases were entered into WebMD symptom checker by both medically trained and nonmedically trained personnel blinded to the diagnosis. Output from the symptom checker, including the number of symptoms, ranking and list of diagnoses, and triage urgency were recorded. The study was conducted on October 13, 2017. Analysis was performed between October 15, 2017, and April 30, 2018.

Main Outcomes and Measures

Accuracy of the top 3 diagnoses generated by the online symptom checker.

Results

The mean (SD) number of symptoms entered was 3.6 (1.6) (range, 1-8). The median (SD) number of diagnoses generated by the symptom checker was 26.8 (21.8) (range, 1-99). The primary diagnosis by the symptom checker was correct in 11 of 42 (26%; 95% CI, 12%-40%) cases. The correct diagnosis was included in the online symptom checker's top 3 diagnoses in 16 of 42 (38%; 95% CI, 25%-56%) cases. The correct diagnosis was not included in the symptom checker's list in 18 of 42 (43%; 95% CI, 32%-63%) cases. Triage urgency based on the top diagnosis was appropriate in 7 of 18 (39%; 95% CI, 14%-64%) emergent cases and 21 of 24 (88%; 95% CI, 73%-100%) nonemergent cases. Interuser variability for the correct diagnosis being in the top 3 listed was at least moderate (Cohen κ = 0.74; 95% CI, 0.54-0.95).

Conclusions and Relevance

The most popular online symptom checker may arrive at the correct clinical diagnosis for ophthalmic conditions, but a substantial proportion of diagnoses may not be captured. These findings suggest that further research to reflect the real-life application of internet diagnostic resources is required.

This cross-sectional study examines the use of internet-based tools in the identification and classification of ophthalmologic symptoms inputted by patients.

Introduction

The accessibility of internet-based resources is increasing globally and searching for health information represents one of the most common uses of the internet.¹ Online symptom checkers (OSCs) are widely used tools that apply computer algorithms to create differential diagnoses based on symptoms input by patient. The OSCs represent an amalgamation of modern accessibility and technology with the most fundamental aspect of clinical medicine: the patient history. As more patients are presenting with self-guided research of symptoms, it is important for ophthalmologists to be aware of the capabilities and limitations of available health information tools. To our knowledge, no study has investigated the utility of OSCs in ophthalmology; these tools represent an unvalidated means of health information delivery.

Methods

In a cross-sectional descriptive study, we generated 42 clinical vignettes from peer-reviewed sources representing the most common ophthalmic diagnoses encountered in clinical practice^2,3,4,5 and validated them with practicing ophthalmologists (eTable in the Supplement). Each clinical vignette was categorized as emergent or nonemergent based on diagnosis per Channa et al.⁵ This study was exempted from research ethics board approval by the Hamilton Integrated Research Ethics Board.

The key symptoms and basic demographic data from these vignettes were distilled and entered in to WebMD⁶ by 2 individuals, 1 medical and 1 nonmedical personnel on October, 13, 2017. Analysis was performed between October 15, 2017, and April 30, 2018. Participants were masked to the correct diagnosis and provided only with the clinical vignette and core symptoms. Output consisting of the inputted symptoms, generated list of diagnoses, and triage urgency of the top diagnoses was recorded.

Results

Of the 42 vignettes, 18 cases were emergent conditions and 24 were nonemergent. The mean (SD) number of symptoms per vignette entered was 3.6 (1.6) (range, 1-8), of which mean (SD) 0.5 (0.8) (range, 0-3) were extraocular. The median (SD) number of diagnoses generated by the OSC was 26.8 (21.8) (range, 1-99). The primary diagnosis by the OSC was correct in 11 of 42 (26%; 95% CI, 12%-40%) cases. The correct diagnosis was included in the OSC’s top 3 diagnoses in 16 of 42 (40%; 95% CI, 25%-56%) cases. The correct diagnosis was not included in the OSCs list in 18 of 42 (43%; 95% CI, 32%-63%) cases. The mean position on the differential list generated by the OSC when the correct diagnosis was listed was 4.7 (8.2) (range, 1-39). The most common primary diagnosis made by the symptom checker was nearsightedness, which was made in 9 cases.

Triage urgency based on the top diagnosis was appropriate in 7 of 18 (39%; 95% CI, 14%-64%) emergent cases and 21 of 24 (88%; 95% CI, 73%-100%) nonemergent cases. Eleven of the 14 cases in which the triage urgency of the primary diagnosis was incorrect would have led to an urgent case being triaged as nonurgent. Interuser variability between the medical and nonmedical personnel for the correct diagnosis being in the top 3 listed was at least moderate based on the lower bound of the 95% CI (Cohen κ = 0.74; 95% CI, 0.54-0.95).

Discussion

Previous studies have highlighted the challenges in delivering and interpreting appropriate health information online.^7,8 An additional layer of complexity is added with the possibility of self-diagnosis through various online tools. In this study, we characterize the current terrain of the most popular online symptom checker regarding its application to ophthalmic symptoms. A previous study by Semigran et al⁹ evaluated 23 symptom checkers using 45 clinical vignettes representing several general medical conditions. They found that, overall, the correct diagnosis was listed first in 34% of evaluations, in the first 3 diagnoses in 51%, and in the top 20 diagnoses in 58%. Other studies have examined OSCs in otolaryngology,¹⁰ orthopedics,¹¹ and plastic surgery.¹² Overall, the reported OSCs have been more accurate in their assessment than we found in our study of ophthalmic conditions. This difference may be because of the nature of ophthalmology, in which a spectrum of diseases share similar clinical manifestations, often necessitating reliance on physical examination and ancillary testing.

Semigran et al⁹ also found the OSCs to be more accurate for nonemergent and common conditions; however, appropriate triage advice was provided in 80% of emergent cases and 55% of nonemergent cases. This difference is likely a result of the risk-averse nature of triage information provided by OSCs that favors directing patients to medical attention rather than observation. However, our study found that most cases of inaccurate triage were a result of emergent conditions being triaged as nonemergent based on the top diagnosis. This finding may reflect the overall lower prevalence of life- and limb-threatening conditions in ophthalmology combined with the less-accurate top diagnosis in OSCs when applied to ophthalmic symptoms. Until the accuracy and information provided by OSCs are improved, there is a risk of unnecessary use of health care services and, conversely, missed opportunities for appropriate intervention when seeking care is delayed.

Strengths and Limitations

Strengths of our study include the standardized data imputation that controlled for several patient- and scenario-dependent variables, a large spectrum of clinical diagnoses, and the fact that the true diagnosis was known. Conversely, this standardization may be viewed as a weakness that does not capture the real-life utility of these tools. The association between educational level, age, technological savviness, anxiety, and other patient factors that affect the ability to decipher the presented diagnoses could not be studied. Generated vignettes were primarily devoid of comorbidities and distractor symptoms, which may have overestimated the accuracy of the OSC. However, these vignettes reflect the type of clinical scenarios used to train physicians in pattern recognition. Although the study examined only a single OSC, WebMD is the most commonly used⁹ and was the most robust in terms of symptom entry and differential generation compared with other OSCs that we examined for ophthalmic conditions. Given the proprietary technology surrounding WebMD and most other OSCs, it is difficult to dissect how the OSC arrives at the list of diagnoses and understand the cause of the inaccuracies. In addition, although only 2 participants were involved in entering symptoms, agreement between the 2 individuals was found. Since this study was conducted, WebMD has redesigned the interface and output of their OSC.

Conclusions

Our evaluation of the utility of the most popular online symptom checker for ophthalmic conditions suggests that, although it is possible to arrive at the correct clinical diagnosis, a substantial proportion of diagnoses may not be captured. This finding reflects an incompleteness of the underlying data set from which ophthalmic diagnoses are drawn and imperfections in the decision-making algorithms that may be used. There is room for improvement in the domain of online symptom checkers for ophthalmic symptoms. Future studies may investigate the accuracy of OSCs compared with ophthalmologists for patients in real-life settings. As the dominion of internet diagnostics expands further in medicine, there may be changes in the way that ophthalmologists practice.

Supplement.

eTable. Ophthalmic Clinical Vignettes

eReferences