Quantifying clinicians’ diagnostic uncertainty when making initial treatment decisions for microbial keratitis

Abstract

Purpose:

There is a need to understand physicians’ diagnostic uncertainty in the initial management of microbial keratitis (MK). This study aimed to understand cornea specialists’ diagnostic uncertainty by establishing risk thresholds for treatment of MK that could be used to inform a decision curve analysis for prediction modeling.

Methods:

A cross-sectional survey of cornea specialists with at least two years clinical experience was conducted. Clinicians provided the percentage risk at which they would always or never treat MK types (bacterial, fungal, herpetic, and amoebic) based on initial ulcer sizes and locations (<2mm² central, <2mm² peripheral, and >8mm² central).

Results:

72 of 99 ophthalmologists participated who were 50% female with an average of 14.7 (standard deviation=10.1) years of experience, 60% in academic practices, and 38% outside the United States (U.S.). Clinicians reported they would “never” and “always” treat a <2mm² central MK infection if the median risk was 0% and 20% for bacterial (interquartile range, IQR=0–5 and 5–50), 4.5% and 27.5% for herpetic (IQR=0–10 and 10–50), 5% and 50% for fungal (IQR=0–10 and 20–75), and 5% and 50.5% for amoebic (IQR=0–20 and 32–80), respectively. Mixed-effects models showed lower thresholds to treat larger and central infections (p<0.001, respectively), and thresholds to always treat differed between MK types for U.S. (p<0.001) but not international clinicians.

Conclusion:

Risk thresholds to treat differed by practice location, and MK types, location, and size. Researchers can use these thresholds to understand when a clinician is uncertain and to create decision support tools to guide clinicians’ treatment decisions.

Introduction

Clinicians are often faced with uncertainty when making diagnostic decisions. A few causes of diagnostic uncertainty include lack of knowledge about a disease, unclear clinical histories, misinterpretation of findings, and limitations of diagnostic testing, to name a few causes.¹ Diagnostic uncertainty impacts the wellness of patients, effectiveness of clinicians, and other aspects of healthcare including utilization, cost, and medical errors.² For eye conditions, delays in diagnosis can contribute to worse vision loss.³ Understanding and quantifying diagnostic uncertainty for specific conditions help to identify areas for improvement regarding high-risk conditions and develop solutions to address the uncertainties.²

Diagnostic uncertainty exists for microbial keratitis (MK) as identifying the underlying causative organism can prove difficult due to overlapping clinical presentations of differing organisms.⁴ Cultures obtained from scrapings are the diagnostic gold standard, but only identify 50% of the causative MK organisms.^5–6 Cultures take days and there are no current in-office point-of-care tests. This uncertainty in identification of the causative organism drives suboptimal care for patients as treatments are often broadly targeted which can increase unnecessary side effects and costs for both the patient and health care system. Timely and accurate diagnosis of the causative organism has the potential to improve outcomes for patients with MK.⁷ Machine-learning prediction algorithms for MK organism type are being explored.^8,9 However, the performance of these algorithms have not been evaluated with respect to the decision that clinicians face. For example, in a case of a corneal ulcer for a contact lens-wearing patient in the United States (U.S.), clinicians will have a low threshold to always treat with an antibacterial agent. In that case, the clinician would have minimal need for a prediction algorithm to recommend antibacterial treatment. Similarly, a U.S. clinician may never treat a small peripheral lesion with an antifungal agent, so, again, there is no need for an algorithm to aid the clinician. Thus, prediction models are most helpful in cases when the clinician is uncertain if they should treat or not treat. Defining uncertainty quantitatively allows researchers to evaluate both performance and value of a diagnostic prediction tool.

Decision Curve Analysis (DCA) is a methodology that provides a framework to assess the clinical usefulness of prediction models by considering the range of risk thresholds for treatment without explicitly assigning costs and benefits to all possible outcomes (i.e., true positives, true negatives, false positives, and false positives) that traditional decision analysis requires.^10–12 DCA can be useful when assessing the range of uncertainty of treatment risk to provide a net benefit to clinicians from evaluations of real-world performances of prediction algorithms.^10,13–15 DCA has been effectively used in other disciplines of medicine such as bladder cancer¹⁶, low birth weight¹⁷, and in the eye condition myopia.¹⁸ Lin and colleagues utilized DCA for evaluating strategies to screen for myopia in school aged children. Children completed three tests, and their data was used to determine both the accuracy and the net benefits of screening strategies.¹⁸ To our knowledge, DCA has not been used in microbial keratitis research to date but has been used for eye conditions such as cataracts and myopia.^18–19 Creating the clinical context to inform ophthalmic prediction modeling will be critical if algorithms are going to be implemented. Thus, the goal of this study is to establish clinicians’ risk thresholds for a specific management decision: when to treat or not treat MK with a specific medication class.

Materials and Methods

The University of Michigan Institutional Review Board reviewed and exempted this study. Survey development included multiple iterative reviews with the research team, including cornea specialists, and underwent internal pilot testing with cornea specialists at University of Michigan. The final survey included questions about the percentage risk thresholds to always or never treat initial MK according to different scenarios. Survey scenarios varied by MK types (bacterial, fungal, herpetic, and acanthamoeba), and sizes and locations (<2mm² central, <2mm² peripheral and >8mm² central). For example, “In a case of a <2mm² central corneal ulcer, if I believed that the patient had a _xx_% risk of bacterial keratitis, I would always treat with antibacterial medications.”

The main outcome measure was the percentage risk threshold determined by the clinicians. Demographic factors of the clinicians were self-reported and included age, gender, race, ethnicity, years of experience, practice type, and practice location. Clinicians were included if they completed cornea training (including fellowship, as applicable to their nation) and had practiced for at least two years. They were intentionally sampled across gender. The survey data were collected and managed using the REDCap electronic data capture tools (Vanderbilt University, Nashville, TN) between March and April 2022. Purposive sampling was conducted by generating a list of cornea specialists to be contacted for potential participation. Emails sent to potential participants included an explanation of the study and a survey participation code. Participants were contacted up to three times via email.

Statistical Analysis:

Demographic and survey responses of the ophthalmologists were summarized with descriptive statistics, including means, standard deviations (SD), medians, interquartile ranges (IQR)s, frequencies, and percentages. Threshold differences between MK types were investigated with Kruskal-Wallis tests followed by Holm-adjusted Dunn tests for post-hoc pairwise comparisons. Linear mixed effects regression models were used to investigate factors associated with percentage risk to always or never treat. Fixed effects included gender, years of experience, practice type, MK location and size, and interaction between practice location and MK type. Random effects included ophthalmologists as random intercepts and MK type as random slopes. Repeated risk threshold measures within a respondent were modeled with an “unstructured” covariance structure. R version 4.1.1 (R Core Team; Vienna, Australia) was used for statistical analysis.

Results

The survey response rate was 72.7% (n=72 of 99 clinicians). One response was excluded from analysis due to inverse responses for treat and never treat risk thresholds, implying a misunderstanding of the question stem. Participants were on average 44.8 years (SD=9.9) with an average of 14.7 (SD=10.1) years of experience and 50% female, 49% Asian, 45% White, 3% Black, and 6.2% Hispanic. There were 60% (n=43) of respondents who practiced in an academic setting. There were 38% of respondents (n=27) from international locations. International locations included 14 countries and included: Argentina, Armenia, Australia, Egypt, Ethiopia, India, Israel, Malaysia, Mexico, Nepal, Netherlands, Singapore, Thailand, and the United Kingdom. Further demographics of study participants are displayed in Table 1.

Table 1.

Demographics and survey responses of Ophthalmologists

Continuous Variable	Mean (SD)	Min, Max	Median (IQR)
Age (years)	44.8 (9.9)	31.0, 77.0	42.0 (37.0, 51.0)
Experience (years)	14.7 (10.1)	2.0, 51.0	11.0 (7.8, 19.0)
Categorical Variable	Frequency (%)
Gender
Female	34 (50.0)
Male	34 (50.0)
Race
Asian	33 (49.3)
White	30 (44.8)
Black/African American	2 (3.0)
Other	2 (3.0)
Ethnicity
Hispanic	4 (6.2)
Non-Hispanic	61 (93.8)
Practice Type
Academic	43 (59.7)
Private	11 (15.3)
Hybrid	15 (20.8)
Other	3 (4.2)
Location
USA	44 (62.0)
International	27 (38.0)

SD, Standard Deviation; IQR, Interquartile Range; USA, the United States of America

Clinicians reported they would never and always treat an initial <2mm² central infection with specific medication types if the median risk were 0% and 20% for bacterial (IQR=0–5 and 5–50), 4.5% and 27.5% for herpetic (IQR=0–10 and 10–50), 5% and 50% for fungal (IQR=0–10 and 20–75), and 5% and 50.55% for amoebic (IQR=0–20 and 32–80), respectively. Reported risk thresholds were significantly lower for bacterial infections compared to fungal (never, p=0.006; always, p<0.001), amoebic (never, p=0.003; always, p<0.001), and herpetic infections (never, p=0.046), and for herpetic compared to amoebic infections (always, p=0.001). Further results for <2mm² peripheral and >8mm² central infection scenarios are displayed in Table 2. All post-hoc pairwise comparisons are provided in Supplemental Table 1.

Table 2.

Ophthalmologists’ responses to “At what perceived risk of bacterial/fungal/herpetic/amoebic keratitis would you never or always treat with antibacterial/antifungal/antiviral/steroid medications?”

	“Never” Treat		“Always” Treat
MK Scenarios	Median (IQR), Mean (SD)	P-value *	Median (IQR), Mean (SD)	P-value *
<2mm2 central
Bacterial	0.0 (0, 5), 4.5 (8.8)	0.002a,b,c	20.0 (5, 50), 29.1 (27.8)	<0.001a,b,f
Fungal	5.0 (0, 10), 9.9 (13.9)		50.0 (20, 75), 48.4 (29.9)
Herpetic	4.5 (0, 10), 9.0 (12.8)		27.5 (10, 50), 35.6 (27.5)
Amoebic	5.0 (0, 20), 12.1 (15.9)		50.5 (32, 80), 55.0 (30.1)
<2mm2 peripheral
Bacterial	1.5 (0, 10), 6.3 (9.1)	0.126	25.0 (10, 56), 35.4 (28.2)	<0.001a,b,c,f
Fungal	8.5 (0, 16), 11.3 (14.5)		50.0 (20, 79), 53.5 (31.7)
Herpetic	5.0 (0, 16), 10.6 (14.1)		40.0 (19, 70), 41.2 (28.7)
Amoebic	5.0 (0, 20), 14.2 (20.1)		60.0 (30, 84), 58.2 (30.7)
>8mm2 central
Bacterial	0.0 (0, 1), 3.4 (8.3)	<0.001a,b,c	10.0 (5, 26), 23.0 (29.3)	<0.001a,b,c,f
Fungal	3.0 (0, 10), 9.0 (13.7)		30.0 (10, 63), 40.0 (29.6)
Herpetic	1.0 (0, 10), 7.7 (12.3)		20.0 (10, 50), 30.2 (25.9)
Amoebic	5.0 (0, 20), 10.7 (14.4)		50.0 (24, 75), 49.0 (29.9)

^*Kruskal-Wallis test

Holm-adjusted Dunn’s tests for post-hoc pairwise comparison showed significant risk threshold differences for:

^aBacterial versus Fungal

^bBacterial versus Herpetic

^cBacterial versus Amoebic

^dFungal versus Herpetic

^eFungal versus Amoebic

^fHerpetic versus Amoebic

SD, Standard Deviation; IQR, Interquartile Range

Figure 1 displays the uncertainty range between which clinicians would always and never treat by MK types and practice location. U.S. clinicians reported the greatest median range of uncertainty (when to never or always treat) for acanthamoeba (53%), followed by fungal (40%), herpetic (13.5%), and bacterial (10%) infections. International clinicians reported the greatest uncertainty range for acanthamoeba (46%), followed by fungal (17%) and herpetic (17%), and bacterial (5%) infections.

Figure 1.

Line plot displaying the median reported percentage risk of microbial keratitis (MK) at which a clinician would never or always treat with corresponding medications, stratified by MK type and the ophthalmologist’s practice location (US, United States; Intl, International).

The results from linear mixed effects models showed risk responses for always treat differed significantly by size, location, and organism type for U.S. clinicians only (Table 3). Specifically, the threshold to treat >8mm² central ulcers was lower by 6.8 percentage points compared to <2mm² central ulcers (p<0.001), and by 12.1 percentage points compared to <2mm² peripheral ulcers (p<0.001). The risk threshold to treat <2mm² central MK was lower by 5.2 percentage points compared to <2mm² peripheral ulcers (p<0.001). U.S. respondents’ threshold to treat bacterial infections was lower by 26.2 percentage points compared to fungal (p<0.001) and by 37.3 percentage points compared to amoebic (p<0.001). Similarly, threshold to never treat >8mm² central was lower by 1.2 percentage points compared to <2mm² central (p=0.04) and by 2.9 percentage points compared to <2mm² peripheral ulcers (p<0.001); threshold of <2mm² central was lower than <2mm² peripheral ulcers by 1.6 percentage points (p=0.01). U.S. respondents reported bacterial infection median risk was lower than fungal by 8.4 percentage points (p=0.001) and amoebic by 10.9 percentage points (p<0.001). Risk was not associated with years of experience, gender, or practice type. Supplemental Figure 1 displays estimates of threshold for MK scenario comparisons by practice locations.

Table 3.

Results of linear mixed effects models estimating the effect of clinician characteristics and microbial keratitis (MK) scenarios on the reported risk threshold to always or never treat with targeted medication.

		“Never” Treat			“Always” Treat
Term	Contrast	Effect	95% CI	Holm-adjusted P-value	Effect	95% CI	Holm-adjusted P-value
Experience (Years)	-	0.1	−0.01, 0.3	0.072	−0.2	−0.7, 0.3	0.46
Gender	Male vs Female	−0.6	−4.0, 2.9	0.754	−7.1	−18.5, 4.3	0.216
MK Size & Location	<2mm2 Central vs >8mm2 Central	1.2	−0.2, 2.6	0.043	6.8	4.5, 9.2	<0.001
	<2mm2 Peripheral vs >8mm2 Central	2.9	1.4, 4.3	<0.001	12.1	9.7, 14.4	<0.001
	<2mm2 Peripheral vs <2mm2 Central	1.6	0.2, 3.1	0.012	5.2	2.8, 7.6	<0.001
MK Type x Practice Location	Comparison between MK Type in US
	F vs B	8.4	2.1, 14.7	0.001	26.2	10.9, 41.5	<0.001
	H vs B	5.4	−0.3, 11.1	0.078	9.3	−1.1, 19.7	0.097
	H vs F	−3.0	−8.4, 2.5	1	−16.9	−31.0, −2.8	0.005
	A vs B	10.9	3.3, 18.4	<0.001	37.3	22.1, 52.6	<0.001
	A vs F	2.5	−3.5, 8.5	1	11.1	2.1, 20.2	0.004
	A vs H	5.4	−0.7, 11.6	0.131	28.0	14.4, 41.6	<0.001
	Comparison between MK Type for Intl.
	F vs B	1.8	−5.9, 9.4	1	4.6	−14.0, 23.2	1
	H vs B	3.6	−3.3, 10.6	1	2.8	−9.8, 15.5	1
	H vs F	1.9	−4.7, 8.5	1	−1.7	−18.9, 15.4	1
	A vs B	4.2	−4.9, 13.4	1	7.2	−11.4, 25.7	1
	A vs F	2.5	−4.8, 9.7	1	2.6	−8.4, 13.6	1
	A vs H	0.6	−6.9, 8.1	1	4.4	−12.2, 20.9	1
	Comparison of Intl. vs US for each MK Type
	B	2.5	−4.2, 9.2	1	20.5	−3.5, 44.5	0.133
	F	−4.1	−15.3, 7.1	1	−1.1	−27.1, 24.9	1
	H	0.7	−9.9, 11.3	1	14.0	−10.3, 38.4	0.978
	A	−4.1	−17.7, 9.5	1	−9.7	−35.1, 15.8	1
	Remaining Comparison
	B Intl. vs F US	−5.9	−14.5, 2.8	0.736	−5.7	−30.2, 18.8	1
	B Intl. vs H US	−2.9	−11.3, 5.5	1	11.2	−12.8, 35.2	1
	B Intl. vs A US	−8.4	−18.3, 1.6	0.198	−16.8	−41.1, 7.5	0.483
	F Intl. vs B US	4.3	−5.3, 13.8	1	25.1	0.2, 50.0	0.037
	F Intl. vs H US	−1.1	−12.0, 9.7	1	15.8	−9.3, 40.9	0.735
	F Intl. vs A US	−6.6	−18.7, 5.5	1	−12.3	−37.9, 13.4	1
	H Intl. vs B US	6.1	−3.0, 15.3	0.736	23.3	−0.7, 47.4	0.05
	H Intl. vs F US	−2.2	−13.0, 8.5	1	−2.9	−27.6, 21.9	1
	H Intl. vs A US	−4.7	−16.5, 7.1	1	−14.0	−38.6, 10.6	0.983
	A Intl. vs B US	6.7	−4.5, 18.0	1	27.7	3.1, 52.3	0.011
	A Intl. vs F US	−1.6	−14.2, 11.0	1	1.5	−24.1, 27.0	1
	A Intl. vs H US	1.3	−11.1, 13.7	1	18.4	−6.4, 43.2	0.343
Practice Type	Private vs Academic	−2.8	−9.4, 3.8	0.49	2.2	−19.3, 23.6	1
	Hybrid vs Academic	−5.7	−12.7, 1.3	0.119	−15.9	−38.7, 7.0	0.377
	Other vs Academic	8.7	−3.0, 20.5	0.14	3.6	−34.6, 41.8	1
	Hybrid vs Private	−2.9	−11.0, 5.2	0.49	−18.0	−44.4, 8.4	0.377
	Other vs Private	11.6	−1.0, 24.2	0.075	1.4	−39.6, 42.5	1
	Other vs Hybrid	14.5	2.3, 26.6	0.012	19.5	−20.3, 59.2	0.746

B=Bacterial; F=Fungal; H= Herpetic; A=Acanthamoeba; Intl=International; US = United States; CI, Confidence Interval

Discussion

Cornea specialists had the lowest threshold to always treat bacterial keratitis stating 20%, 25% and 10% (IQR=5–50, 10–56, and 5–26) for each scenario (<2mm² central, <2mm² peripheral, and >8mm² central), followed by herpetic with 27.5%, 40%, and 20% (IQR=20–75, 20–79, and 10–63), fungal with 50%, 50% and 30% (IQR=10–50, 19–70, and 10–50), and acanthamoeba with 50.5%, 60% and 50% (IQR= 32–80, 30–84, and 24–75). Not surprisingly, clinicians had a lower risk threshold for large central ulcers compared to small peripheral and small central. Uncertainty was also found to vary by presumed risk of organism type and clinical parameters. Organisms risk assessments differed as well, where it was observed that risk thresholds were significantly lower for bacterial infections, specifically for >8mm² central “always” and “never” treat. This was also observed for the “never” treat <2mm² central scenario, while it was only significant for amoebic and fungal in the <2mm² central scenario and <2mm² peripheral scenario “always” treat scenarios. Differences were observed amongst uncertainty ranges for ophthalmologists practicing in the U.S. and internationally, specifically acanthamoeba had the greatest range of uncertainty (53% vs. 46%), followed by fungal (40% vs. 17%), herpetic (13.5% vs. 17%), and bacterial (10% vs. 5%) MK types. In addition, ophthalmologists practicing internationally had a greater range of uncertainty as compared to those practicing in the U.S.

This survey focused on scenarios of various MK types. Eye clinicians’ ability to predict the organism causing MK infections has wide range of accuracy.^20–22 Dalmon and colleagues reported that cornea specialists examining fungal and bacterial ulcers were able to accurately predict gram stain 46% of the time, while only identifying the genus and species 25% and 10% of the time, respectively.²² An additional study including 421 ophthalmologists by Xu and colleagues found that ophthalmologists classifying infectious keratitis had a 49.7% ± 11.5% (range: 20.00%–86.67%) of diagnostic accuracy.²⁰ Clinicians have been able to identify acanthamoeba infections (positive predictive value of 89%, 95% CI: 52% to 100%) by observing a ring infiltrate on examination.²³

In this study, fungal and acanthamoeba MK types had the greatest risk thresholds for initial treatment. This could possibly be because acanthamoeba and fungal infections are treated with very toxic medications as compared to bacterial infections. In addition to the medications being very toxic they are needed for prolonged periods of time.²⁴ This study also observed that international clinicians have differed from U.S. clinicians potentially due to different barriers to care and supported by previous research suggesting differences in practice patterns by region.²⁵ Peeler and colleagues found that ophthalmologists from Nepal recommended that Nepal patients with anterior segment disease come back sooner than the recommendations by ophthalmologists in the U.S.²⁵ Internationally, areas such as Nepal and India, may have a higher percentage of fungal or atypical corneal ulcers. Because of these additional exposures, ophthalmologists practicing in these countries may have more comfort in treating and recognizing these infections, thus they may have a lower risk threshold for initial treatment of these MK types.

Current advancements in the identification of organisms causing MK include the use of machine-learning prediction.^26–28 Evaluating the performance of an algorithm with real-world context is possible using DCA. The findings from this research demonstrate that between identified risk thresholds lies uncertainty where a diagnostic tool could be beneficial to inform patient treatment decisions. Other future research could explore the reasons for the clinicians assigned specific risk thresholds. They could also explore the patient and decision-making factors to understand why risk thresholds differ among clinicians.

Limitations of this study include utilizing purposive sampling which was chosen to optimize diversity in responses and increase response rate but may cause selection bias due to its non-random nature. Participants may have understood the survey questions differently, as one participant “reversed” their responses assigning a higher threshold to never treat instead of a higher threshold to always treat. Not all countries were represented equally as recruitment could not be universally implemented. Lastly, specific patient demographics for each clinician were not known, potentially affecting the risk threshold choices by the clinicians.

In summary, these findings suggest that ophthalmologists’ risk thresholds to always or never treat MK infections vary by organism types, size, and location. Between these risk thresholds to always and never treat MK infections is the range of diagnostic uncertainty. There is still a need for clinical judgment, but this diagnostic uncertainty range is where a decision curve analysis could be beneficial for showing value of prediction modeling and area of future exploration.

Supplementary Material

Supplemental Figure 1

Supplemental Figure 1. Comparison of estimated marginal mean effect of risk threshold between US and international respondents.

Supplemental Table 1

Supplemental Table 1. Holm-adjusted Dunn Test Pairwise Comparisons for at which a clinician would never or always treat with corresponding medications, for all MK scenarios.

Corneal Ulcer Study Group:

Guillermo Amescua, Masih Ahmed, Zaina Al-Mohtaseb, Diana Alvarez-Melloni, Sejal Amin, Menen Ayalew, Ashwin Balasubramanian, Winston Chamberlain, Matilda Chan, Elsie Chan, Meenu Chaudhary, Thomas Chia, James Chodosh, Josephine Christy, John Clements, John Dart, Mohammad Dastjerdi, Matthew Denny, Mohamed Elghobaier, Chris Estopinal, Preethika Gandhi, Miles F. Greenwald, Nikhil Gokhale, Natalie Hernandez, Anna Hovakimyan, Frank Hwang, David Hwang, Tomas Jaeschke, Vishal Jhanji, Faris Karas, Carol Karp, Lakshmi Kattana, Jeremy Keenan, Sumitra Khandelwal, Tyson Kim, Ellen Koo, Aaleya Koreishi, Jennifer Li, Tom Lietman, Marian Macsai, Jaime Martinez, Jod Mehta, Michael Mimouni, Adam Moss, Afshan Nanji, Nathan Nataneli, Jennifer Nussbaumer, Vasudha Panday, Stephen Pflugfelder, Sayali Pradhan, N. Venkatesh Prajna, Naveen Rao, Travis Redd, Satya Reddy, Alfonso Sabater, Julie Schallhorn, Gerami Seitzman, Ruti Sella, Neha Shaik, Sankalp Singh Sharma, Nakul Shekhawat, David Spokes, Alan Sugar, Audrey Talley Rostov, Napaporn Tananuvat, Chulaluck Tangmonkongvoragul, Rahul Tonk, Sonal Tuli, Phit Upaphong, Suvitha Vairavan, Bart Van Dooren, Manoj Vasudevan, Angela Verkade, Meraf Wolle, Maria Woodward, Rachel Wozniak, Choong YeanYaw.

Data Availability:

Dr Woodward had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. The dataset generated and analyzed during the current study are not available.