Helping GPs to extrapolate guideline recommendations to patients for whom there are no explicit recommendations, through the visualization of drug properties. The example of AntibioHelp® in bacterial diseases

Rosy Tsopra; Karima Sedki; Mélanie Courtine; Hector Falcoff; Antoine De Beco; Ronni Madar; Frédéric Mechaï; Jean-Baptiste Lamy

doi:10.1093/jamia/ocz057

. 2019 May 11;26(10):1010–1019. doi: 10.1093/jamia/ocz057

Helping GPs to extrapolate guideline recommendations to patients for whom there are no explicit recommendations, through the visualization of drug properties. The example of AntibioHelp^® in bacterial diseases

Rosy Tsopra ^1,^2,^✉, Karima Sedki ³, Mélanie Courtine ³, Hector Falcoff ⁴, Antoine De Beco ⁴, Ronni Madar ¹, Frédéric Mechaï ^6,⁷, Jean-Baptiste Lamy ³

PMCID: PMC7647204 PMID: 31077275

Abstract

Introduction

Clinical decision support systems (CDSS) implementing clinical practice guidelines (CPGs) have 2 main limitations: they target only patients for whom CPGs provide explicit recommendations, and their rationale may be difficult to understand. These 2 limitations result in poor CDSS adoption. We designed AntibioHelp^® as a CDSS for antibiotic treatment. It displays the recommended and nonrecommended antibiotics, together with their properties, weighted by degree of importance as outlined in the CPGs. The aim of this study was to determine whether AntibioHelp^® could increase the confidence of general practitioners (GPs) in CPG recommendations and help them to extrapolate guidelines to patients for whom CPGs provide no explicit recommendations.

Materials and Methods

We carried out a 2-stage crossover study in which GPs responded to clinical cases using CPG recommendations either alone or with explanations displayed through AntibioHelp^®. We compared error rates, confidence levels, and response times.

Results

We included 64 GPs. When no explicit recommendation existed for a particular situation, AntibioHelp^® significantly decreased the error rate (−41%, P value = 6x10⁻¹³), and significantly increased GP confidence (+8%, P value = .02). This CDSS was considered to be usable by GPs (SUS score = 64), despite a longer interaction time (+9–22 seconds). By contrast, AntibioHelp^® had no significant effect if there was an explicit recommendation.

Discussion/Conclusion

The visualization of weighted antibiotic properties helps GPs to extrapolate recommendations to patients for whom CPGs provide no explicit recommendations. It also increases GP confidence in their prescriptions for these patients. Further evaluations are required to determine the impact of AntibioHelp^® on antibiotic prescriptions in real clinical practice.

Keywords: clinical decision support system, clinical practice guidelines, antibiotics, infectious diseases, primary care, visualization

BACKGROUND AND SIGNIFICANCE

Inappropriate antibiotic prescriptions may have harmful consequences, such as increasing antimicrobial resistance,¹ or causing patient complications—or even death.² It also has an impact on health care system costs.²

The prescription of antibiotics is particularly complex in primary care because general practitioners (GPs) usually have to make decisions empirically, that is, without knowing the identity of the causal agent.³^,⁴ They also have to consider other factors such as bacterial resistance rates⁵ and patient preferences.⁶

National health authorities provide narrative clinical practice guidelines (CPGs) to help GPs to prescribe the most appropriate antibiotic.⁷ CPGs include evidence-based recommendations in the form of “clinical situation/antibiotic” associations, which are then implemented in clinical decision support systems (CDSS) in the form of “If/then” rules⁸^,⁹ (eg, “If acute uncomplicated cystitis, then prescribe fosfomycin trometamol”).³ However, despite the existence of CDSS, 30% of antibiotic prescriptions for outpatients remain inappropriate.^10–12 This may be due to a lack of GP confidence in the recommendations¹³ and the difficulties GPs face when trying to extrapolate the recommendations to real patients.¹⁴^,¹⁵ Indeed, CPGs provide no explicit recommendations for many clinical situations,¹⁶ and GPs must then combine knowledge from multiple sources and their own experience to guide their decisions.

We designed a CDSS, AntibioHelp^®, to overcome these shortcomings. AntibioHelp^® displays both the recommended and nonrecommended antibiotics, together with their properties, weighted by degree of importance. We hypothesized that the visualization of these weighted antibiotic properties would help GPs to have greater confidence in the recommendations due to a better understanding of the underlying rationale and extrapolate the recommendations to patients for whom CPGs provide no explicit recommendations but only partial guidance. For example, CPG experts recommend fosfomycin trometamol in uncomplicated cystitis in women because it has the following properties: 1) it is not contraindicated, 2) it is microbiologically active against Escherichia coli, and 3) it causes few adverse effects. If fosfomycin cannot be prescribed (eg, due to contraindications), the GP could consider another antibiotic with similar properties, such as pivmecillinam. Furthermore, it has been shown in other studies that visualization is a potentially valuable approach to decision support.¹⁷

The aim of this study was to evaluate whether an interface displaying the antibiotics and their weighted properties could increase the confidence of GPs in CPG recommendations and help GPs to extrapolate recommendations to patients for whom the CPGs provide no explicit recommendations.

Previous work - design of AntibioHelp^®

We previously identified the properties used by CPG experts to decide which antibiotics to recommend¹⁸^,¹⁹ and the degree of importance (or weight) of each property.²⁰ This information is not explicit in CPGs. We first had to analyze the pieces of justifications occasionally given by CPG experts to explain their recommendations (eg, “fosfomycin should be preferred because it causes few side effects”). This analysis showed that experts establish their recommendations on the basis of 11 antibiotic properties (eg, “few side effects”).¹⁸^,¹⁹ Some of these properties are necessary, and only antibiotics that satisfy these properties can be prescribed (eg, absence of contraindication). Others are preference properties, and antibiotics that satisfy these properties are preferred over those that do not (eg, antibiotics with few side effects are preferred over those with more side effects).

We built a knowledge base describing these 11 properties for 50 antibiotics, 11 infectious diseases, and 21 patient profiles.²⁰ Each property is Boolean (true or false) and its value can also be unknown. The “true” value always corresponds to an advantage, that is, an argument in favor of the prescription of the antibiotic. The values depend on the antibiotic but also the causal bacteria, patient profile, and the infectious disease involved. The knowledge base was structured as an OWL 2.0 ontology, including 144 038 resource description framework triples describing 5696 classes, 19 properties, and 34 483 axioms. It belongs to the ALC(D) family of description logics.

For a given clinical situation (ie, a given infectious disease in a patient with a given profile), the antibiotics and their properties can be modeled as sets of elements: the antibiotics are the elements, and we can define sets for each value of each property (eg, the set of antibiotics that are contraindicated, the set of antibiotics with few adverse effects, and the set of antibiotics with many adverse effects). For each necessary property prop, we consider a single set, Set_{Necess, prop}, including all antibiotics that do not satisfy this property (ie, the value is false or unknown). For each preference property prop, we consider 2 sets: Set_{PrefAdv, prop}, including all antibiotics for which the property is true, and Set_{PrefDis, prop}, including all antibiotics for which the property is false.

For the visualization of these sets, we used a recently developed technique for set visualization, called rainbow boxes.^21–23 In the resulting interface (Figure 1), AntibioHelp^®, all the antibiotics are displayed in columns, and the sets describing the properties are displayed in colored rectangular boxes. Each box covers the columns of the antibiotics belonging to a given set (ie, sharing a given value for a given property). When the columns covered by a box are not contiguous, holes are present in the box: the box is split into several parts connected by a thin horizontal line (eg, in Figure 1, the box labeled “Moderate efficacy” has 1 hole of size 2 corresponding to the columns “ceftriaxone” and “amoxicillin clavulanate”). Antibiotics are separated into 2 groups:

Figure 1. — Visualization of the weighted drug properties with rainbow boxes. Example of otitis in a child over the age of 6 years, without allergy at the top and with allergy to beta-lactam and sulfonamide at the bottom.

The antibiotics on the right, with a dark gray column header, are those that should not be prescribed because they do not have the necessary properties. The sets Set_{Necess, prop}, describing the necessary properties not satisfied, are displayed in light red boxes.
The antibiotics on the left, with a light gray column header, are those that could be prescribed because they have all the necessary properties. The sets Set_{PrefAdv, prop} and Set_{PrefDis, prop} describing the preference properties are displayed in a light green box if the property is advantageous (ie, true), or a light orange box if the property is disadvantageous (ie, false). No box is displayed if the value of the property is unknown. Box height is proportional to the weight attributed by the experts in the CPGs. These weights were learnt in a preference model,²⁰ using a metaheuristic algorithm.²⁴ For each preference property prop, 2 weights, Weight_{PrefAdv, prop} and Weight_{PrefDis, prop}, are considered for Set_{PrefAdv, prop} and Set_{PrefDis, prop}, respectively. The preference model ranks the antibiotics that have the necessary property. The model computes a score (the utility function u), corresponding to a linear combination of the property values for antibiotic a:
$u (a) = \sum_{prop} {\begin{matrix} Weigh t_{\Pr ef Adv, prop} \\ Weigh t_{\Pr ef Dis, prop} \\ 0 \end{matrix} \begin{matrix} if a \in Se t_{Pref Adv, prop} \\ if a \in Se t_{Pref Dis, prop} \\ otherwise \end{matrix}$

Antibiotics with a higher score are ranked as better than those with a lower score. The metaheuristic algorithm finds the optimal values for Weight_{PrefAdv, prop} and Weight_{PrefDis, prop}, that is, those ranking the antibiotics in the same order as in the CPGs. Properties with higher weights are considered to be more important than those with lower weights. The preference model is generic and covers the 11 infectious diseases considered.

Table 1 lists the 11 properties, their type (necessary or preference) and their weights (for preference properties).

Table 1.

Properties used by CPG experts to determine which antibiotics to recommend. Preference properties have weights derived from a preference model learnt from CPGs with a metaheuristic algorithm (20). For each preference property prop, 2 weights, Weight_{PrefAdv, prop} and Weight_{PrefDis, prop}, were learnt for Set_{PrefAdv, prop} (ie, the set including all antibiotics for which the property is true), and Set_{PrefDis, prop} (ie, the set including all antibiotics for which the property is false), respectively

Properties	Type	Weight_{PrefAdv, prop}	Weight_{PrefDis, prop}
Naturally active against the causal bacterium	Necessary	–	–
Probably active against the causal bacterium	Necessary	–	–
Clinical efficacy proven in the disease	Necessary	–	–
Absence of contraindication for the patient	Necessary	–	–
Convenient protocol	Preference	4	7
Nonprecious class	Preference	2	2
Absence of serious and frequent side effects	Preference	2	5
High level of efficacy	Preference	2	2
Narrow antibacterial spectrum	Preference	2	2
Low level of ecological adverse effects	Preference	2	3
Acceptable taste	Preference	2	3

Open in a new tab

Abbreviation: CPG, clinical practice guideline.

In addition, AntibioHelp^® displays:

The rank of recommendation of the antibiotic in the CPGs, indicated as a number displayed below the name of the antibiotic.
Detailed information about antibiotics and their properties is provided in an interactive information bubble on demand, when the cursor is moved over the name of the antibiotic (mouseover). Icons are also associated with each antibiotic property to facilitate the reading of the interface.
Brand names or generic names of antibiotics, which can be requested by ticking a checkbox at the bottom left of the screen.
The pharmacological class of each antibiotic, indicated by a brightly colored bar above the antibiotic name. At the bottom right of the screen, 6 checkboxes can be used to hide the antibiotics belonging to a given pharmacological class (eg, if the patient is allergic to the antibiotics of the class concerned).

The interface is generated automatically for all clinical situations modeled in the ontology. For example, Figure 1 displays the antibiotics and their properties for otitis in children over the age of 6 years. In this case, 6 antibiotics have all the necessary properties and could therefore be prescribed. At a glance, we can see that 1 of these antibiotics, amoxicillin, has the greatest total height of green boxes and the smallest total height of orange boxes. It is, therefore, the most appropriate antibiotic in this clinical situation. Furthermore, in the column header, we can see that this is the top-ranking antibiotic recommended in the CPGs.

This interface can also be helpful for extrapolating recommendations to patients for whom CPG recommendations do not apply due to contraindications or allergy (eg, for otitis in a child over the age of 6 years with allergies to beta-lactams and sulfonamide). In such cases, the GP can tick the checkboxes at the bottom of the screen to gray out the antibiotics belonging to the classes to which the patient is allergic (the beta-lactam and sulfonamide classes in this case). The GP can then see, at a glance, which antibiotics have the necessary properties and their preference properties and can, therefore, decide which antibiotic is the most appropriate. Here, only 1 antibiotic, pristinamycin, has all the necessary properties (Figure 1). This antibiotic is, therefore, the most appropriate.

MATERIALS AND METHODS

We investigated whether the visualization of weighted antibiotic properties via AntibioHelp^® could increase GP confidence in CPG recommendations and help GPs to extrapolate the recommendations to patients for whom CPGs provided no explicit recommendations.

To achieve this goal, we asked GPs to respond to clinical cases on the basis of CPG recommendations either alone or with the assistance of AntibioHelp^®. We tested both 1) clinical cases for which CPGs provided explicit recommendations, to determine whether the interface increased GP confidence; and 2) clinical cases for which CPGs provided no explicit recommendations to determine whether GPs extrapolated recommendations better with the interface. GPs were reminded of the CPG recommendations relating to the disease of the clinical case.

Evaluation design

We carried out a 2-stage crossover study in which the participating GPs were asked to respond to clinical cases presented in a random order with and without access to AntibioHelp^® (Figure 2). The evaluation was performed online with Firefox or Chrome.

Figure 2. — Diagram of the crossover study (SUS: System Usability Scale).

GP recruitment

Participants were contacted by e-mail via medical associations. For involvement in the study, the GPs had to be in training or practicing in primary care and they had to have an internet connection. The evaluation was free and anonymous.

Constitution of clinical cases

We first extracted 57 clinical situations from 2 CPGs for respiratory infections. We then randomly chose 2 clinical situations for set A, and 2 for set B. Randomization was stratified by type of disease (eg, otitis) and patient profile. For each clinical situation, we built 2 clinical cases: a simple clinical case for which a recommendation was provided in the CPGs and a complex clinical case for which there was no explicit recommendation. For example, for the clinical situation “otitis in a child over the age of 6 years,” we built 2 clinical cases: 1) “otitis in an 8-year-old child without allergy” for which a recommendation exists and 2) “otitis in an 8-year-old child with beta-lactam and sulfonamide allergy” for which no explicit recommendation was provided in the CPGs.

This process yielded 2 sets of clinical cases: A and B. Set A contained 2 simple and 2 complex cases of otitis and pneumonia. Set B contained 2 simple and 2 complex cases of sinusitis and pharyngitis.

For training, we also built 2 simple and 2 complex clinical cases of cystitis and pyelonephritis. The results for these cases are not analyzed here.

Crossover design

GPs were randomly assigned to 2 groups:

In group 1, GPs first considered 2 of the training cases and set A without AntibioHelp^®, and they then considered the other 2 training cases and set B with AntibioHelp^®.
In group 2, GPs first considered 2 of the training cases and set A with AntibioHelp^®, and they then considered the other 2 training cases and set B without AntibioHelp^®.

In both groups, when GPs had to answer without AntibioHelp^®, they were shown a recap of the CPG recommendations relating to the disease concerned in the form of a simple table displaying the recommended antibiotics and their rank in the CPG recommendations (eg, uncomplicated acute cystitis → fosfomycin trometamol in rank 1).

After using AntibioHelp^®, GPs completed the System Usability Scale (SUS). SUS is a 10-item scale for evaluating usability.²⁵^,²⁶ For each item, the degree of agreement is assessed with a 5-point Likert scale. A total SUS score between 0 and 100 was then calculated.

At the end of the evaluation, GPs were asked whether they preferred to visualize the weighted drug properties, and they were invited to write optional comments in free text.

Measurement criteria

We measured 5 criteria:

Error rate was measured by determining the number of inaccurate responses for each clinical case. A gold standard was derived by a medical doctor (RT) and a specialist in antibiotics (FM). For simple clinical cases, the prescription of any antibiotic recommended in rank 1 in the CPG was considered correct. For complex clinical cases, the prescription of any antibiotic that could safely be used to treat the patient was considered correct.
Confidence level was measured on a 7-point scale²⁷: totally unsure (2% certainty), unsure (10% certainty), doubtful (25% certainty), more or less sure (50% certainty), sure enough (75% certainty), sure (90% certainty), very sure (98% certainty).
Response time was measured as the time taken to give a response for the clinical case.
Perceived usability was measured by calculating the SUS Score.
Preference was measured by determining the proportion of GPs preferring to have access to AntibioHelp^®.

Statistical analysis

The error rates were compared in Fisher’s tests. The impact of GP characteristics (such as age, sex, time in practice, last medical training in antibiotics, and frequency of external resource use), and their interaction with access to the interface (with/without AntibioHelp^®) were assessed with a generalized linear model. Confidence levels and response times were compared in student’s t tests. R V.3.5.1 statistical software was used.

RESULTS

We included 64 GPs in the study: 59% were women, and almost half were under the age of 40 years and had been in practice for less than 20 years. More than half of the GPs declared using an external resource for antibiotic prescription more than once per week (Table 2).

Table 2.

Sociodemographic characteristics of the GPs

	Group 1 (Percentage % (number))	Group 2 (Percentage % (number))	Total (Percentage % (number))
Sex
Male	13% (8/64)	28% (18/64)	41% (26/64)
Female	34% (22/64)	25% (16/64)	59% (38/64)
Age
20-29 years old	2% (1/64)	6% (4/64)	8% (5/64)
30-39 years old	23% (15/64)	14% (9/64)	38% (24/64)
40-49 years old	5% (3/64)	6% (4/64)	11% (7/64)
50-59 years old	8% (5/64)	9% (6/64)	17% (11/64)
Over 60 years old	9% (6/64)	17% (11/64)	27% (17/64)
Time in practice
In training	3% (2/64)	3% (2/64)	6% (4/64)
0-4 years	6% (4/64)	8% (5/64)	14% (9/64)
5-9 years	11% (7/64)	9% (6/64)	20% (13/64)
10-19 years	11% (7/64)	5% (3/64)	16% (10/64)
20-29 years	5% (3/64)	9% (6/64)	14% (9/64)
Over 30 years	11% (7/64)	19% (12/64)	30% (19/64)
Last medical training in antibiotics
Less than 1 year ago	9% (6/64)	6% (4/64)	16% (10/64)
Between 2 and 5 years ago	23% (15/64)	25% (16/64)	48% (31/64)
Between 6 and 10 years ago	8% (5/64)	14% (9/64)	22% (14/64)
Over 11 years ago	6% (4/64)	8% (5/64)	14% (9/64)
Frequency of external resource use for antibiotic prescriptions
Never	0% (0/64)	2% (1/64)	2% (1/64)
Rarely (<once/month)	2% (1/64)	3% (2/64)	5% (3/64)
Sometimes (1 to 3 times/month)	13% (8/64)	14% (9/64)	27% (17/64)
Frequently (>once/week)	19% (12/64)	20% (13/64)	39% (25/64)
Almost all the time	14% (9/64)	14% (9/64)	28% (18/64)

Open in a new tab

Abbreviation: GP, general practitioner.

Error rate

For simple clinical cases, the error rate was 19.5% without AntibioHelp^® vs 20.3% with AntibioHelp^®. This difference was not significant (P value = 1, Fisher’s test).

For complex clinical cases, the error rate was 91% without AntibioHelp^® vs 50% with AntibioHelp^®. This difference was significant (P value = 6x10⁻¹³, Fisher’s test).

The generalized linear model showed that 3 factors had a significant impact on error rate: 1) access to the interface (P = 6x10⁻⁶), 2) the complexity of the clinical case (P < 3x10⁻¹⁶), and 3) the interaction of these 2 factors (P = 3x10⁻⁷). These results are consistent with the results of Fisher’s exact tests. Other factors (age, sex, time in practice, last medical training in antibiotics, and frequency of external resource use) had no significant impact on error rate.

Thus, when no explicit recommendations were available, the visualization of the weighted antibiotic properties significantly improved antibiotic prescription by GPs (Figure 3).

Figure 3. — Error rate for simple and complex clinical cases by type of disease. The error rate was 24%–38% higher for clinical cases relating to pneumonia and pharyngitis than for other diseases. The missing bar for otitis and sinusitis corresponds to an error rate of 0%.

GP confidence

For simple clinical cases, GP confidence level was 86% without AntibioHelp^® vs 87% with AntibioHelp^®. This difference was not significant (P value = .5, Student’s t test).

For complex clinical cases, GP confidence level was 63% without AntibioHelp^® vs 71% with AntibioHelp^®. This difference was significant (P value = .02, Student’s t test).

Thus, when no explicit recommendations were available, the visualization of the weighted antibiotic properties significantly improved GP confidence level (Figure 4).

Figure 4. — Confidence level for simple and complex clinical cases by type of disease.

Response time

For simple clinical cases, the response time was 37 seconds without AntibioHelp^®versus 46 seconds with AntibioHelp^®. This difference was significant (P value = .0007, Student’s t test).

For complex clinical cases, the response time was 61 seconds without AntibioHelp^®versus 83 seconds with AntibioHelp^®. This difference was significant (P value = .001, Student’s t test).

Thus, the visualization of weighted antibiotic properties significantly increased the response time (Figure 5).

Usability

The mean SUS score for AntibioHelp^® was 64. The perceived usability of the interface was therefore “good,” according to the scale proposed by Bangor et al.²⁸^,²⁹ The interface seemed to be easy to use, and no technical support was required for 77% of GPs (Figure 6).

Figure 6. — Distribution of answers to each question of the SUS (System Usability Scale).

GP preferences and free comments

Overall, 58% of GPs reported preferring having access to the weighted antibiotic properties, vs 42% who preferred not to have such access.

Some GPs wrote free comments highlighting the usefulness of AntibioHelp^® for the management of complex patients. It was reported by 1 GP that the interface could be used to study antibiotic properties. Another 2 explained that AntibioHelp^® makes it easier to understand antibiotic choice and to take into account the clinical reasoning behind guidelines. Conversely, 1 GP said that in clinical practice, he/she only needed to see which antibiotic is recommended—not its properties. A few of the GPs said that they appreciated the content displayed (eg, bacterial susceptibility), whereas others found the tool too complex. Several GPs suggested that AntibioHelp^® could be improved by adding antibiotic doses, the names of the causal bacteria, filters for pregnancy and renal failure, and by removing some antibiotics that were displayed but not used in primary care.

DISCUSSION

Our results show that the visualization of weighted antibiotic properties helped GPs to extrapolate recommendations to patients for whom CPGs provide no explicit recommendations. It also increased the confidence of GPs in their prescriptions for these patients. Conversely, the visualization of antibiotic properties had no significant effect when explicit recommendations were available. The presentation of weighted antibiotic properties in the form of rainbow boxes was considered usable by GPs despite the longer interaction time required (an additional 9–22 seconds).

Methodology used for evaluation

We conducted a rigorous study by randomly assigning GPs and clinical cases and conducting a 2-period crossover study. We also used 2 different sets of clinical cases to prevent memory effects (GPs beginning the second part of the study might recall the responses they gave during the first part of the study). Our study design was also subject to several limitations: 1) the GPs were quite young, probably because of the mode of evaluation (we carried out an online evaluation which may have selected for the youngest GPs who were the most comfortable with computers); 2) training in the use of AntibioHelp^® was limited to an online tutorial, with no possibility of asking questions; 3) the study was carried out online, so the conditions of the evaluation were not, therefore, controlled. Nevertheless, we deliberately chose to perform a study of this type because it made it possible to include more GPs (GPs are more likely to participate in studies if they can perform the evaluation anytime and anywhere), and it was suitable for evaluation of prototypes under development.