Abstract
Computer-based decision support systems (CDSSs) are currently mostly reminder systems. However, the effectiveness of such systems to modify physician behavior is not always observed. We assume that this approach is appropriate when physicians think they know how to prescribe and consider they don’t need to be helped, i.e. for simple clinical cases. On the opposite, on-demand approaches allowing for flexibility in the interpretation of patient conditions are more appropriate for more complex cases, e.g. in chronic disease management. ASTI is a CDSS operating in two modes, a critiquing mode working as a reminder-based system and a user-initiated guiding mode. Using a clinical case complexity score, a pre/post-intervention experiment with 10 GPs and 15 cases of hypertensive patients has been performed. Preliminary results tend to indicate that reminder-based interaction is appropriate for simple cases and that physicians are willing to use on-demand systems as clinical situations become complex, making both modes complementary.
Introduction
Clinical practice guidelines (CPGs) are commonly being developed and disseminated to improve the care received by patients. However, there is still considerable variation in the effectiveness of guidelines to change the behavior of clinicians.
Computer-based guideline systems can be effective in increasing physician compliance with recommendations. However, failure to improve adherence using computer-based strategies was reported in numerous studies1,2 addressing the issue of design factors responsible for the success or the failure of computer-based guideline intervention strategies. Though most studies fail to describe the systems used, CDDSs are usually based on the display of onscreen reminders and alerts automatically provided from patient-specific data. This seems to be “neither necessary nor sufficient” to ensure a positive impact on physician compliance rate with recommendations.2
More recently, new studies confirmed electronic reminders benefit on routine health maintenance items and preventive care tests in ambulatory settings including vaccinations and cancer screening. But clinicians’ needs for information to support decisions in the management of chronic diseases such as diabetes or hypertension cannot be solved through the display of reminders.3 On the contrary, “on-demand systems”, using a structured knowledge-base that the user may read or browse, thus allowing for flexibility in guideline interpretation, are currently being considered.4 On-line information retrieval systems (Up-to-Date, Micromedex, Quick Clinical, etc.) seem to have won clinicians’ favor. For the authors, the success of such systems is based on the underlying usage paradigm that allows physicians to actively participate to a decision-making process that respects their clinical expertise, instead of only being data-suppliers in reminder-based approaches. Beyond being well accepted by physicians, on-demand systems have proven to significantly improved the accuracy of clinicians’ therapeutic decisions.5
Our hypothesis is thus that two different situations have to be distinguished in primary care general practice. On the one side, there are clinical cases that general practionners (GPs) know how to solve, although they might be wrong and propose inappropriate therapies leading to unconscious medical errors. These cases are usually “simple” cases and GPs are convinced they do not need to be helped. For such simple cases, reminder-based interaction is both mandatory (the display of the recommended therapy may allow the GP to improve her prescription) and technically feasible (only few patient characteristics have to be considered to decide). On the other side, there are clinical cases for which GPs may seek for up-to-date evidence to support decision-making. These cases are usually “more complex” cases for which GPs easily admit they lack knowledge about what is the recommended treatment to prescribe and conscienciously accept that either they don’t know how to treat the patient or that their prescription might be suboptimal. In such cases, GPs would deliberately use guidance systems.
ASTI6 is a prototype guideline-based CDSS, applied to therapeutic prescribing in primary care, following this hypothesis. ASTI can be used according to two modalities. Dedicated to simple cases, the “critiquing mode” operates classically as a background process to control physician’s orders. Working as a reminder-based system, ASTI critiquing mode issues alerts when the physician’s order differs from guideline recommendations. On the contrary, the “guiding mode” of ASTI operates on demand when the GP needs support to determine the drug prescription of a complex case. ASTI has been first applied to the management of hypertension. The knowledge bases have been built from the 1999 Canadian recommendations. This paper presents a preliminary evaluation of ASTI. The aim is to check that (i) reminder-based interaction increases physician compliance with recommendations on simple cases, (ii) on-demand approaches increase physician compliance with recommendations on complex cases, and (iii) that both operating modalities are complementary in practice.
Background
A flexible approach to guideline modeling has been proposed by Tu and Musen.7 The underlying assumption is that there exist two different classes of guidelines: on the one side, one-shot guidelines considered as “consultation guidelines” that specify actions and decisions whose consequences are not being tracked over time. These guidelines usually cover the management of patients with simple conditions, i.e. large vaccination campaigns or management of acute diseases for which the data needed to provide patient-specific recommendations involve only current data. On the other side, the management of chronic diseases relies on “management guidelines”, more complex, that model decisions and actions that lead to dependent changes in patient states over time. For instance, Prodigy phase I and II3 operated as if GPs’ medical practice could be only modeled by consultation guidelines. There was one guideline per diagnosis with possible patient situations within that diagnosis organized into scenarios. Automatic choice of a scenario from the findings recorded in the patient electronic medical record (EMR) resulted in a list of possible actions displayed in a reminder-based interaction. When evaluated, Prodigy II proved to be technically competent at acute diseases. However, when applied to the management of patients with complex conditions, recommendations issued by Prodigy II were often judged inadequate. This distinction between consultation and management guidelines has to be connected with the distinction between simple and complex clinical cases.
If consultation guidelines can be efficiently represented as if-then-else statements for decision making (Arden syntax, Prodigy phase I and II, etc.), the complexity of management guidelines is better handled by task-based models.8 Many dedicated formalisms based on task networks have been developed aiming to support automated CPG execution. However, if management guidelines describe what should be the right strategy, i.e. the recommended ordered sequence of actions/treatments for a given patient, it is always a theoretical strategy that should be adapted because the disease evolves over time as well as the patient response to treatments. For instance, the Prodigy III model, related to EON, formalizes the guideline content as a network consisting of scenarios, action steps and subguidelines. However, scenarios, expected to provide easy access points into the guideline, are high level views of patient states that do not integrate detailed patient-specific therapeutic history (past treatments, tolerance, efficiency, etc.).
Because the representation in a computer-processable format of management guideline content has intrinsic limitations (formalization of all possible patient conditions is untractable), fully automated medical reasoning processes cannot provide accurate recommendations. Some flexibility in interpreting guidelines as well as patient information is indeed required for CDSSs to gain in effectiveness and thus in physician acceptability. Because classical formal approaches can hardly account for such flexibility, less formal approaches have been proposed to provide physicians with guidance. Guideline knowledge is structured in a way a user could retrieve patient-specific recommendations more easily than within texts. Browsing and reading such structured guideline representations, the physician becomes a mediator of patient information which does not need to be strictly coded. The OncoDoc system8 has been developed to promote these principles. It relies on a knowledge base, formally structured, through which a user navigates according to the informal description of a given patient to get patient-specific recommendations.
ASTI has been developed according to the assumption that both consultation and management guidelines are used in GPs’ daily medical practice. As only therapeutic decisions established for clinical situations covered by the CPG can be criticized, the knowledge base used by the critiquing mode solely formalized the guideline content. However, only simple patient conditions are described in the CPG, since recommendations are provided for the choice of initial therapy for hypertensive patient suffering from only one complication in addition to hypertension. In this way, recommendations are similar to consultation guidelines. They have been modeled as decision rules in the if-then format. On the contrary, the guiding mode of ASTI provide therapeutic options for any cases, whether they are covered by the CPG or not. Therapeutic options relying on professional agreement have thus been added and are displayed as advices (with the corresponding grade of evidence, D) when evidence-based recommendations are missing, e.g. in the case of multiple-condition patients or for second, or higher, lines of treatment. Thus, advices provided by the guiding mode are similar to management guidelines and have been formalized as a two-level decision tree.10
Material and method
We carried on a study following the pre-post intervention experimental design to measure the impact of ASTI upon prescription behavior. Another aim was to check our working assumption based on the hypothesis that GPs would use the critiquing mode for simple cases and the guiding mode for complex cases. This is based on the following formal characterization of clinical case complexity.
Quantification of clinical case complexity
We formalized the complexity of a clinical case c by a complexity score that accounts for two dimensions which participate in the overall case complexity: (i) the patient’s clinical situation and (ii) the characterization of her therapeutic history.
Clinical complexity, denoted CC(c), measures the complexity of the clinical situation defined as the number of diseases or risk factors associated to hypertension, as well as known contraindications to therapeutic classes, e.g. CC(uncomplicated hypertension) = 0, CC(hypertension+diabetes) = 1, CC(hypertension+diabetes+ACE-i contraindicated) = 2.
Therapeutic complexity, denoted TC(c), measures the complexity of the patient’s therapeutic history, defined as the number of treatments previously administered in terms of therapeutic classes, e.g. TC(never treated) = 0, TC(ACE-i) = 1, TC(ACE-i,ACE-i+thiazides) = 2.
Each kind of complexity participates independently to the overall case complexity and the complexity score CS of a clinical case c is simply defined as the sum: CS(c) = CC(c) + TC(c)
Considering that cases presenting 2 elements of complexity are moderately difficult to manage, three complexity levels were defined as follows:
| Low | Medium | High |
|---|---|---|
| CS< 2 | CS =2 | CS> 2 |
Material
Fifteen clinical cases of hypertension have been extracted and anonymized from actual GPs’ medical records of hypertensive patients. The distribution of complexity in the sample was approximately balanced. The 15 cases were dispatched in three 5-case sets as reported in table 1.
Table 1.
Distribution of case complexity and average complexity score for case sets
|
Class |
|||||
|---|---|---|---|---|---|
| n | Low | Medium | High | Av. CS | |
| Set 1 | 5 | 4 | – | 1 | 1.4 |
| Set 2 | 5 | – | 2 | 3 | 3.2 |
| Set 3 | 5 | 1 | 2 | 2 | 2,4 |
| All | 15 | 5 | 4 | 6 | 2,3 |
Protocol
The protocol followed a pre-post intervention design. The study was conducted on one day with 10 volunteering GPs, all éO users (éO is the EMR used in ASTI). Each GP was self-controlled.
Pre-intervention phase without ASTI
Every participant p received the paper version of the 15 clinical cases. For each clinical case c, each participant was invited to solve the case, and write down the drug order, noted di(p,c), she decided to prescribe. During this step, participants did not have access to any computer or supplementary material. They were not aware they will have to review the same cases later in the post-intervention phase.
Post-intervention phase with ASTI
After a presentation of the ASTI project and a tutorial on how to use the system, participants were asked to review the same 15 cases with the support of ASTI. At this point, they did not have access to their initial decisions. For each case, each participant had thus to select the corresponding patient record in éO and use ASTI to decide the treatment. Three different steps were defined to control the use of both the critiquing and the guiding modes of ASTI, according to the prior clustering of the 15 cases in 3 sets of 5 cases.
Step 1: critiquing mode use. Participants had to use the critiquing mode alone on the cases of set 1. They entered their drug prescription in the order entry module of éO. Depending on the status of the entered prescription with respect to recommendations, alerts with suggestions may be displayed or not. The order could be modified as many times as necessary until their final decision is reached.
Step 2: guiding mode use. Participants had to use the guiding mode alone on the cases of set 2. Using the EMR to collect patient data, they navigated in the knowledge base to get ASTI recommendations. Then they had to make a decision that may follow or not the system’s recommendations.
Step 3: free use. Participants were free to use the ASTI system to help them make the best therapeutic decision on the cases of set 3. They could use either the mode that seemed the most appropriate to their needs, or both modes. In this step, they were asked to specify which mode they used, and in case they used both modes, which one they used first.
Whatever the step, for each case c, each participant p had to write down the drug treatment she prescribed, denoted df(p,c).
Measured variables
Drug commercial names given in GPs’ orders were substituted by their corresponding pharmaco-therapeutic classes to reach the abstraction level required by recommendations (e.g. ACE inhibitors, thiazides, etc.). These abstract prescriptions are denoted Di(p,c), respectively Df(p,c), for initial, resp. final decisions. Each participant being self-controlled, the comparison of Df(p,c) and Di(p,c) is used to measure decision changes, and thus the impact of the system upon GPs’ prescription behavior. The comparison of D•(p,c) and R(c), where R(c) represents ASTI’s recommendations considered as the reference, is used to measure physician compliance with guideline recommendations without and with ASTI. In step 3, we collected the first mode GPs used to support their final decisions as their preferred mode.
Results
We collected 150 paired therapeutic decisions (Di(p,c),Df(p,c)), i.e. 15 clinical cases analyzed by 10 GPs without and with ASTI.
Use of critiquing mode on simple cases (step 1)
Initial physician compliance with recommendations is 32% as reported in table 2. The final compliance rate is 66%. GPs changed their mind in 54% of the cases. Among these cases, the change was positive (from a non compliant decision to a compliant decision) in 66% of cases. There was only one negative change (from a compliant decision to a non compliant decision). As previously mentioned, set 1 included one case of high complexity. For this case, initial and final compliance rates were null, although some GPs changed their mind when using ASTI. This case had a complexity score of 4 and concerned a hypertensive patient suffering of renal failure, currently under a bitherapy of ACE inhibitors and loop diuretics, ACE inhibitors being not tolerated. For the remaining 4 simple cases, compliance increased from 40% to 82% after the use of ASTI.
Table 2.
Measures of compliance and impact
|
Compliance |
|||
|---|---|---|---|
| Step | Impact | without ASTI | with ASTI |
| Step 1 | 54% | 32% | 66% |
| Step 2 | 80% | 10% | 44% |
Use of guiding mode on complex cases (step 2)
Initial compliance is 10% whereas final compliance is 44% (Table 2). GPs changed their mind in 80% of the cases, including 42% of positive changes and 0 negative change. For the 2 medium complexity cases, compliance increased from 25% to 90%. For the other 3 high complexity cases, compliance was initially null and reached 23% after using ASTI.
Free use of ASTI (step 3)
Compliance level increased from 30% to 76%. Set 3 is composed of cases of various complexity (low: 1, medium: 2, high: 2). Cumulatively, the critiquing mode is more used than the guiding mode for the low complexity case (60% vs 40%). It must be noticed that for this simple case, 2 participants did not use ASTI at all. For medium complexity cases, the frequency of use of the critiquing mode did not change (60%), but, the frequency of guiding mode use increased to 80%, meaning some GPs used both modes. For highly complex cases, critiquing mode use felt to 35% while guiding mode use jumped to 95%.
This empirical study has been conducted on a small sample and is not a test of ASTI in practice. Although statistical data analysis cannot be performed, it seems that participants chose a given scenario of system interaction according to the case complexity. This is not invalidated when looking at the first mode used according to case complexity as illustrated by figure 1 where dark bars represent the proportion of guiding mode uses and light grey bars those of critiquing mode uses. The critiquing mode is preferably used as the first choice of decision support for low and medium complexity cases with frequencies of respectively 60% and 55%. This means GPs consider they have sufficient knowledge and a good understanding of the cases to be able to propose an acceptable prescription. As for the guiding mode, it is nearly never used (20%) for the low complexity case. However, it becomes more and more used as first support for medium (45%) and, especially, for high complexity cases (85%). In these later situations, the critiquing mode is not much used (15%). This suggests GPs might think they do not have sufficient knowledge to prescribe and are ready to actively search for guideline support.
Figure 1.
Distribution of preferred mode usage according to case complexity in step 3
Conclusion
We have performed a preliminary evaluation of ASTI to check if the particular design of the system based on a critiquing mode and a guiding mode could be effective to increase physician compliance with recommendations.
This study have strong limitations (small sample size, in vitro although with real clinical cases, not controlled), but the first results are quite encouraging. The critiquing mode used on simple cases increased physician compliance with recommendations as well as the guiding mode used on medium and high complexity cases. However, in this latter case, the level of compliance rates after using ASTI remains low, indicating that GPs are reluctant to adhere to the proposed recommendations. This could be explained by the fact that, for high levels of complexity, recommendations mostly rely on professional agreement and are not evidence-based. Finally, the distribution of ASTI usage in the free step demonstrates the complementarity of both modes, the critiquing mode being used for simple cases and the guiding mode being used for complex cases. A real size controlled trial is planned to confirm these preliminary results.
References
- 1.Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes. JAMA. 1998;280:1339–46. doi: 10.1001/jama.280.15.1339. [DOI] [PubMed] [Google Scholar]
- 2.Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. JAMIA. 1999;6(2):104–14. doi: 10.1136/jamia.1999.0060104. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Johnson PD, Tu S, Booth N, Sugden B, Purves IN. Using scenarios in chronic disease management guidelines for primary care. J Am Med Inform Assoc. 2000;7(suppl):389–93. [PMC free article] [PubMed] [Google Scholar]
- 4.Eccles M, McColl E, Steel N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerized evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomized control trial. BMJ. 2002;325:941–7. doi: 10.1136/bmj.325.7370.941. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Westbrook JI, Coiera EW, Gosling AS. Do on-line information retrieval systems help experienced clinicians answer clinical questions? J Am Med Inform Assoc. 2005;12(3):315–21. doi: 10.1197/jamia.M1717. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Séroussi B, Bouaud J, Dréau H, Falcoff H, Riou C, Joubert M, et al. ASTI, a guideline-based drug-ordering system for primary care. In: Patel VL, Rogers R, Haux R, editors. Medinfo. 2001. pp. 528–32. [PubMed] [Google Scholar]
- 7.Tu SW, Musen MA. A flexible approach to guideline modeling. J Am Med Inform Assoc. 1999;6(suppl):420–4. [PMC free article] [PubMed] [Google Scholar]
- 8.Peleg M, Tu SW, Bury J, Ciccarese P, Fox J, Greenes RA, et al. Comparing computer-interpretable guideline models: a case-study approach. J Am Med Inform Assoc. 2003;10(1):52–68. doi: 10.1197/jamia.M1135. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Séroussi B, Bouaud J, Antoine EC. OncoDoc, a successful experiment of computer-supported guideline development and implementation in the treatment of breast cancer. Artif Intell Med. 2001;22(1):43–64. doi: 10.1016/s0933-3657(00)00099-3. [DOI] [PubMed] [Google Scholar]
- 10.Séroussi B, Bouaud J, Chatellier G, Venot A. Development of computerized guidelines for the management of chronic diseases allowing to position any patient within recommended therapeutic strategies. In: Fieschi M, Coiera E, Li YCJ, editors. Medinfo. 2004. pp. 154–8. [PubMed] [Google Scholar]