Revisiting the EBM decision model to formalize non-compliance with computerized CPGs: results in the management of breast cancer with OncoDoc2

Abstract

In 2002, Haynes et al. founded a prescriptive model of evidence-based medicine based on the patient’s clinical state, her preferences, and research evidence, clinical expertise synthesizing the other three components. Revisiting this model of medical decision making, we propose a descriptive model introducing clinicians’ preferences and formalize four reasons of non-compliance with clinical practice guidelines (CPGs). The approach has been applied to breast cancer management decisions taken by multidisciplinary staff meetings (MSMs) at the Tenon hospital, Paris, France, while using a clinical decision support system (CDSS): OncoDoc2. 1,889 MSM decisions have been recorded [February 2007–October 2009]. The compliance rate with CPGs was measured at 91.0%. Non-compliant decisions are mainly “MSM choices” (39.1%) and “particular cases” (34.9%). “Practice evolution” and “patient choices” are less frequent (12.4% and 11.2%). Even with a CDSS, a 100% compliance rate cannot be reached because particular cases fall outside CPGs and borderline cases need to be interpreted by clinicians.

Introduction

Medicine is becoming increasingly complex. Due to the explosion of medical knowledge, health care providers have indeed great difficulties to update their practice at the pace of the state of the art evolution. As a consequence, old, out-of-date, medical practices may persist leading to significant clinical practice variations that do not seem legitimate, and sometimes to medical errors. Besides, in order to provide optimal and up-to-date care, physicians are expected to follow evidence-based medicine (EBM) principles for their decisions. The EBM notion appeared in the ’90s and was defined as the explicit, judicious, and conscientious use of current best evidence from health care research in decisions about the care of individuals and populations.^[1] Around year 2000, the EBM practice model has evolved to recognize the importance of the shared decision that respects patient’s preferences and values and to emphasize clinicians’ expertise. Haynes et al.^[2] modelled the EBM decision according to 4 dimensions: (i) clinical state and circumstances, (ii) patients’ preferences and actions, (iii) research evidence and (iv) the clinical expertise which role has been expanded and appears as a bridge connecting the first 3 dimensions to make the best individualized decision. However, practicing EBM remains an idealized view of medicine, which is difficult to implement in daily practice.^[3]

In order to promote best practices, clinical practice guidelines (CPGs) have been developed as practical tools summarizing the state of the art at a given time to help health care professionals implement EBM. CPGs aim at improving the quality of care by reducing inappropriate variations, producing optimal patient outcomes, and promoting cost-effective practices. CPGs are textual information resources in which practice recommendations for peculiar clinical situations are graded according to their level of scientific evidence although they may also be based on professional agreement. However, the sole dissemination of textual documents has nearly no impact on clinicians’ behavior. Many studies have shown that guideline-based clinical decision support systems (CDSSs) can be effective in increasing physician compliance with CPGs.^[4,5] However, while many CDSSs have been developped, very few are implemented in routine practice. Many obtacles to their adoption remain and the factors of success are not well understood. Criticisms expressed concern CDSSs but may also be related to guidelines contents.^[6,7]

We have developped OncoDoc2,^[8] a CDSS on breast cancer management based on a local reference guideline (CancerEst). OncoDoc2 has been used in a before/after study and improved the compliance rate of multidisciplinary staff meetings (MSMs) with CancerEst CPGs.^[9] However, despite the use of a CDSS, clinicians did not systematically follow the system recommendations and the compliance rate reached 93%. Following the study, OncoDoc2 has been routinely used in senology MSMs of the Tenon Hospital (Paris, France) for nearly 3 years. The compliance rate in routine use has been measured at 91%.^[10]

The objective of this paper is to propose a description of the reasons that lead clinicians not to comply with OncoDoc2 recommendations by revisiting the 4-dimension conceptual model of EBM decision of Haynes et al. Non-compliant decisions collected in Tenon’s breast cancer MSMs have been used to illustrate the categories of non-compliance and to assess the distribution of MSM non-compliant clinical decisions.

Evolution of Evidence-Based Medicine Decision Models

The notion of evidence-based medicine was first introduced by Sackett et al.^[1] as a (more) scientific approach to practicing medicine. The first aim of EBM was to take into account evidences provided by clinical research to make decisions for individual patients. Although the term has been widely disseminated among health care professionals, it is not clear, from the EBM promoters themselves, whether the notion itself has been always well understood. More than 10 years after its birth, many practitioners consider EBM as a theoretical model of medicine, which it is, difficult and time consuming to implement in routine practice.^[3] In 2008, for instance, Hay et al.^[11] mentionned a gap between EBM and actual clinical practice. Despite the large dissemination of CPGs, expected to integrate EBM recommendations, practice variations are still observed.

The very notion of EBM has evolved over time and models have been proposed to account for dimensions that where previously hidden or implicit in order to make them explicit in the EBM decision process. A first step was to recognize the role of patient’s preferences and values and consider that every medical decisions should be shared and agreed by both the patient and the clinician (see for instance Guyatt et al.^[12]). In 1996, a 3-component model accounting for the patient’s preferences, the seminal search for research evidence, and the clinical expertise of the clinician has been proposed. Behind this latter term laid many skills like assessing the patient’s state and problems, searching relevant research evidence, and incorporate the patient’s preferences.

Haynes et al.^[2] provide a brief summary of the EBM evolution until 2002, specifically the recognition that finding the best evidence is not sufficient for delivering appropriate care. In this paper, authors proposed a 4-component prescriptive EBM model (Figure 1) which distinguished explicitely patient information including the factual “clinical state” of the patient and “circumstances” which are contextual information that could indirectly influence the patient management. For the authors, this “clinical state and circumstances” component is the main one every decision should be based on. “Patients’ preferences and actions” is another component that should be dealt with in selecting the relevant options “research evidence” provides. “Clinical expertise” is now a surrounding component of the decision and has an expanded role. It represents the clinician’s skills in assessing the other 3 components and synthesizing them to provide an appropriate decision for the patient. However, according to the authors, such clinical expertise may benefit from the clinician’s experience but excludes her own preferences.

Figure 1:

2002 EBM decision model from Haynes et al.^[2]

EBM remains currently hardly implemented^[11] although attempts have been made to improve the way each component could be better assessed or treated so that the notion could be stabilized on a consensual definition. For instance, Hay et al.^[11] have added practitioners’ own experiences which had been first neglected. These experiences should be better valued as part of the clinical expertise and patients’ clinical state assessement. With respects to the search for the best research evidence, it is acknowledged that finding the best trade-off between competing recommendations is not easy since they might not have the same level of evidence.^[13] As a response, others authors promote a methodological approach and a model for explicitly grading every clinical research results, making a difference between the level of evidence and the strength of recommendations, mentionning benefits and potential harm of applying the recommendation. This systematic grading of recommendations is intended to help clinicians in selecting appropriate recommendations for their patients.

The Guideline-Based CDSS OncoDoc2

OncoDoc^[8] is a guideline-based CDSS providing patient-specific recommendations in the management of breast cancer. In order to harmonize their practices, cancer specialists of CancerEst (the union of 4 university hospitals of the Eastern Paris) have drawn up local reference guidelines for the management of several cancers including non-metastatic breast cancer. OncoDoc2, a new version of OncoDoc, has been developed to account for CancerEst CPGs.

OncoDoc2 has been developed according to the documentary paradigm of decision support.^[8] This approach allows physicians to contextualize both guideline medical knowledge and patient information thus improving a flexible use of guidelines for any given patient and optimizing the patient-specificity of the recommended therapeutic propositions. Like in automatically executable guideline systems, OncoDoc2 relies on a formalized knowledge base (KB) structured as a decision tree. But unlike these systems, OncoDoc2’s KB is designed to be browsed, thus delegating the control of knowledge execution to the physician user. From the root of the decision tree, the physician is asked to characterize her patient clinical profile by clicking at each tree level of the KB to select the appropriate value of decision nodes (medical history, clinical examination, pathology results, etc.). While navigating through the KB, all instantiated patient features are collected to incrementally build a patient dataset which corresponds to the “formal patient” derived from the actual patient. When completing the navigation, the physician finally comes to the treatment plans recommended for the “formal patient” which correspond to the patient-centered recommendations of the CPGs.

In accordance with the French “Cancer Plan” measures, therapeutic decisions for cancer patients should be taken by multidisciplinary staff meetings according to guidelines that may be refined as local or more preferably regional CPGs. At the Tenon hospital, a university hospital located in Paris, France, breast cancer MSMs occur on a weekly basis. All medical specialists involved in breast cancer management (surgeons, radiologists, oncologists, radiotherapists, pathologists, oncogeneticists, etc.) are attending the meeting. This is the place where physicians conjointly elaborate patient-specific best care plans. OncoDoc2 was used during MSMs at the time and place decisions were made. The screen was video-projected so that all participants could see the system recommendations. In case the final MSM therapeutic decision is one of OncoDoc2 propositions, it is automatically inserted in the MSM decision window with the compliant status colored in green. In case of non-compliance, the MSM therapeutic decision has to be entered, which automatically changes the compliance status in orange for the decision. Deviations had to be justified (Figure 2).

Figure 2:

Recording the reason of non-compliance with CancerEst CPGs.

Revisiting the EBM Decision Model proposed by Haynes et al.

From Research Evidence to CPGs: Strong Evidence and Weak Consensus

The medical state of the art can be considered as the amount of information that should be processed in order to optimize the quality of patients management. It relies on a published medical knowledge corpus made of either scientific studies which results are characterized by a strength (A, B, C) to represent actual EBM practices, or consensus-based opinions of experts when evidence is missing (grade D). Usually CPGs propose a synthesis of the state of the art for the management of a given disease. This synthesis may be completed or locally customized to represent regional CPGs.

Due to the constant evolution of medicine, new scientific results are regularly published, and a growing number of alternatives with different benefit and risk profiles exist for many conditions. Thus, CPGs developed at t have to be updated at t + T. Figure 3 depicts the evolution of medical knowledge over time. We represent CPGs as either EBM recommendations located in the internal circle, or consensus-based opinions located in the second layer. Between t and t + T, knowledge update could affect information of all strengths of evidence to either produce new EBM knowledge and new consensus-based opinions, or transform previous consensus-based opinions into EBM knowledge.

Figure 3:

Evolution over time of the amount of EBM recommendations and consensus-based opinions.

From Patients’ Preferences to Patients’ and Clinicians’ Preferences: Emergence of the Shared Decision

In agreement with the model for evidence-based decisions proposed by Haynes (Figure 1), we consider that, depending on the information they have, their degree of aversion to risk and their personal values, patients may express preferences on their treatment options. Besides, clinicians’ preferences (as distinct from clinical expertise) often play an important role in the final decision. When faced to similar patients, clinicians may institute different treatments according to their preferences. This has often been considered as the reason for practice variations in managing similar cases. Another explanation is that “similar” patients are in fact different, and based on their experience, clinicians consider that these different patients should be managed differently.

Our model acknowledges that both patients’ preferences and clinicians’ preferences should be considered to describe actual decisions. Figure 4 represents patients’ preferences and clinicians’ preferences as two circles. As a shared decision, the final decision should be accepted by both actors, thus belonging to the intersection of the two areas. In our application, the clinician has to be replaced by the group of MSM clinicians.

Figure 4:

Intersection of patients’ and clinicians’ preferences to represent the shared decision.

Patient Clinical State: Selection of Criteria

Patient clinical state is usually resumed as a set of criteria. These criteria describe the patient’s history, physical findings, diagnostic tests, and circumstances, to characterize the different diseases the patient may suffer from since actual patients are polypathological and suffer from numerous chronic diseases. However, CPGs provide recommendations for the diagnosis and management of a single given disease (i.e. CPGs for the management of hypertension, CPGs for the diagnosis of diabetes, etc.). As a consequence, when considering a given pathology and thus given CPGs, only some of the patient criteria are involved. Any clinical decision concerning a given patient for a given disease has to take into account relevant disease-specific patient’s data and should be based on the currently available disease-specific CPGs. Thus, for a given patient, a guideline-compliant clinical decision belongs to the intersection of the area inferred by patient’s data and the domain that represents CPGs (see figure 5). Any clinical decision outside this intersection does not comply with CPGs for this patient. In the case CPGs formalization is complete and patient’s condition coding is exhaustive, this intersection would represent the set of guideline-compliant recommendations that a CDSS should automatically provide for this patient.

Figure 5:

Intersection of clinical state and medical knowledge to represent guideline-based patient-specific decisions.

Consolidated View of the Decision Process

For practical purposes, guideline-based patient-specific clinical decision have to match the shared decision constraint and belong to the area colored in green in figure 6. Non-compliant clinical decisions are the complementary clinical decisions within the shared decision area (in red in the figure). It is in this latter space that we have categorized the reasons of MSM clinical decision non-compliance.

Figure 6:

Consolidation of the different views.

Categorization of Reasons for Non-Compliance

We propose 4 main reasons of non-compliance with CPGs in the management of breast cancer. These 4 reasons are depicted in figure 7. To make them clearly readable, we have represented two additional processes involved in the generation of non-compliance. First, we have taken into account the evolution of medical knowledge between two consecutive updates of CPGs (see figure 3). Thus physicians, following the last scientific publications, may chose to adopt precocious results to improve the quality of the management of their patients. This is represented by an additional external layer to CPGs circles. Second, the analysis of patient criteria involved in the clinical decision shows that some of them are quantitative variables defined with a threshold value (tumor size > or < 2 cm, age > or < 35 years, margins of resection > or < 2 mm, etc). When quantitative criteria and threshold values are identical, the whole picture of the patient may be re-interpreted by clinicians to assess how to classify the criteria. Thus we have distinguished an internal layer of the circle “Clinical state and circumstance” to represent the interpreted patient’s state. The 4 reasons of non-compliance are defined as follow:

• Practice evolution: Following the publication of new scientific results, some parts of CPGs may be obsolete and provide sub-optimal management strategies. So do CDSS that rely on CPGs before they are updated. Thus, MSM clinicians do not follow CDSS propositions and thus CPGs although their clinical decision is in intention compliant with the current state of the art. In fact, if CPGs and CDSS were instantly updated, the decision would comply with CPGs. Non-compliant decisions due to the evolution of medical practices (in blue in the figure) are located at the intersection of the layer that illustrates the evolution of medical knowledge and the circle that corresponds to the patient state (actual and interpreted) within the area of patient-clinician shared decision.

• Particular case: The patient has some specific features not covered by CPGs (breast cancer in men, breast cancer during pregnancy, breast cancer with BRCA1 or BRCA2 mutations, etc). These specific features are included in the area representing the actual patient state but they are not covered by CPGs. Besides, they are not the result of clinicians interpretation of the patient condition. Non-compliant clinical decisions with regard to particular cases (in pink in the figure) are thus located in the actual patient state circle, but outside the current medical knowledge zone, within the area of patient-clinician shared decision.

• MSM choice: On the basis of their collective expertise, MSM clinicians may consider that the treatment plan strictly recommended by CPGs is not the best option for a given patient and they may decide on best alternative treatment options. In this case, MSM clinicians have interpreted the actual patient state, arbitrarily reducing or increasing some of her features. The “new” patient state is outside the actual patient state area, and the corresponding non-compliant clinical decisions with regard to MSM preferences (in orange in the figure) is outside the current medical knowledge zone, within the area of patient-clinician shared decision.

• Patient choice: The patient may refuse the recommended treatment plan and express her preference for another treatment option (e.g. preference for a radical surgery and choice of a mastectomy instead of the recommended lumpectomy, or on the contrary, rejection of the recommended mastectomy and choice of a lumpectomy). In this case, patient features are not taken into account in the clinical decision. Non-compliant clinical decisions with regard to patient preferences (in yellow in the figure) may correspond to CPGs treatment plans in agreement with the state of the art, but with no relation with the described clinical state and circumstances. However, they belong to the patient-clinician shared decision area.

Figure 7:

Model of non-compliant clinical decisions.

Results

New Oncodoc2 Interface to Collect Reasons of Non-Compliance

We have developed a new interface to help clinicians input the justification of their decisions when their decisions were non-compliant with CPGs according to the 4 main reasons previously described (Figure 2). Plain text comments may also be collected in a dedicated field.

MSM compliance with CancerEst CPGs

Between February 2007 and October 2009, 1,889 MSM clinical decisions have been recorded while routinely using OncoDoc2 during MSMs. Among them, 1,720 clinical decisions were compliant with CancerEst CPGs, thus a compliance rate of 91.0%.

Distribution of Non-Compliant Decisions

Among the 1,889 clinical decisions recorded, 9.0%, thus 169, did not comply with CPGs although MSM clinicians were using OncoDoc2 that provided them with the recommended patient-specific treatment plan. Among the 169 decisions, figure 8 depicts the distribution of non-compliance according to the categories of non-compliance we have proposed.

• 12.4% of non-compliant decisions (21) are related to practice evolution not yet taken into account in CancerEst CPGs and thus in OncoDoc2. This essentially concerns the evolution of the indications for the sentinel lymph node dissection, an alternative to standard axillary lymph node dissection, initially indicated for tumor smaller than 2 cm, and which indication has been extended to tumor size of 3 and even 4 cm.

• More than one-third of non-compliance (34.9% or 50 clinical decisions) correspond to particular cases not covered by CancerEst CPGs and thus by OncoDoc2. It concerns patients with breast cancer and BRCA1 or BRCA2 mutations for who mastectomy should be preferred whatever the tumor size (thus the recommended surgery is mastectomy even if the tumor size is smaller than 2 cm), elderly patients for who depending on general state of health and comorbidities, hormone therapy may be recommended, instead of chemotherapy or even surgery, breast cancer in men or during pregnancy (no sentinel lymph node dissection to avoid radiation).

• 39.1% of non-compliant decisions (66) correspond to MSM choices mainly based on either an overestimation of the risk leading to more aggressive treatments than recommended (e.g. extra radiotherapy, lymph node dissection instead of sentinel node dissection) or an underestimation of the risk leading to less aggressive treatments than recommended (e.g. lumpectomy instead of mastectomy, hormonotherapy instead of chemotherapy) especially, but not only, when patient data correspond to threshold values of decision parameters (tumor size, margins of resection).

• 11.2% of non-compliant decisions (19) are explained by patient choices, e.g. chemotherapy refusal.

• In the last 4 cases, non-compliant decisions have been classified as “other reasons”. It concerns non-therapeutic decisions (the clip left during the mammotome biopsy has not been found during the lumpectomy, no decision of hormone therapy until knowing hormone receptor status, etc.).

Figure 8:

Distribution of non-compliant decisions according to the 4 categories proposed (n = 169)

Discussion and Conclusion

We have extended the model of EBM clinical decisions proposed by Haynes (Figure 1) to account for actual decisions, introducing 4 reasons of non-compliance with CPGs. The distribution of these reasons has been evaluated from the routine use of OncoDoc2 in MSMs of the Tenon hospital, Paris, France, over a period of 29 months. The high compliance rate observed during the initial before/after study of OncoDoc2^[9] was maintained. However, even with a guideline-based CDSS, we still observed non-compliant MSM clinical decisions.

The use of OncoDoc2 helps MSM clinicians to formalize the patient case description while following the questions asked within the navigation. The “check-list” effect allows for the extensive characterization of the patient clinical state and the building of the appropriate formal patient equivalent. Once the navigation is completed, OncoDoc2 states what are the matching CancerEst recommendations. Thus, using OncoDoc2 automatically compute the intersection of both clinical state and current medical knowledge corpus areas (see figure 5) to provide the patient-specific guideline-based treatment plan. However, the adoption of one of OncoDoc2’s recommendations by MSM clinicians is not systematic. It relies on i) the quality of the displayed recommendations both in absolute and relatively to the quality of OncoDoc2 use and on ii) patients’ and clinicians’ preferences (see figure 4). Since the use of OncoDoc2 has been controlled, the quality of the navigations performed and the right interpretation of clinical cases can be assumed. Thus, the non-adoption of CDSS-generated recommendations cannot rely on the bad quality of recommendations relatively to the bad quality of data input in the CDSS.

Concerning absolute quality of CPGs, we have to consider that CPGs are the synthesis of the medical state of the art in the diagnosis/management of a given disease at a given time. However, as soon as they are published, because medical knowledge is permanently evolving, CPGs are out of date. Thus MSM clinicians may decide to non-comply with CPGs although their decision is compliant with the state of the art. Evolution of practice occurred in 12.4% of non-compliant decisions and should sign the dating process of the CDSS. Such cases would disappear once CPGs are updated. Ideally, acknowledged evolution of practices should trigger the update of both CPGs and CDSS knowledge base. However, experience shows that decisions considered as “evolution of practice” were not systematically taken into account for CPG revision, and that prior recommended attitudes were kept as the standard in guidelines.

The other side of CPGs absolute quality concerns the frequency a patient is not covered by CPGs and considered as a particular case. Particular cases might be handled theoretically by locally customized CPGs and CDSSs. However, apart from frequent cases for which there are alternative therapeutic options, this search for completeness would be an endless task, and the answers given without evidence. On the other hand, evolution of medical knowledge should bring, in the future, answers to specific patient cases considered today as particular. Although OncoDoc2 proposes solutions to some particular situations such as, for instance, how to handle contraindications to lumpectomy, or chemotherapy, particulars cases account for a high percentage of about 35% of non-compliant MSM decisions.

When taking into account preferences of both actors of the shared decision, we made the difference between MSM clinicians and patients. The part due to “patient preferences” can neither be anticipated nor reduced. The patient choice represents 11.2% in our study. It is by nature unpredictable although often observed behavioral patterns could be, like particular cases, anticipated. The rational for ”MSM choice” is a touchy point. This reason represents about 40% of non-compliant decisions. Clinicians know they deviate from their own CPGs, but consider, at that moment and for that specific patient, that they are making the best decision. It seems it is observed especially in borderline cases for which interpretation is not completely straightforward. When data is close to threshold values (age, tumor size, margins of resection, etc.), while patient conditions fall into CPGs scenarios, MSM clinicians may assess the benefit/risk ratio of treatment options and decide not to comply with what CPGs would strictly recommend. These cases delimit the last, restrained, domain where clinician free will and experience are preserved over regulated care. Cognitive qualitative observational studies of these decisions should be done to better understand such situations.

This experiment illustrates that even with a guideline-based CDSS, a 100% compliance rate with CPGs is non accessible because particular cases do not fall into CPGs and borderline cases need to be interpreted by clinicians, at least for the management of breast cancer patients. The asymptotic rate value might be different according to domain and CPGs. For instance, we developped other CDDSs like “ASTI guiding mode” based on therapeutic management guidelines of chronic diseases (AHT, dyslipemia, atrial fibrillation) for general practionners and try to understand barriers to CPG compliance.^[14] But, since they have not yet been routinely used, we don’t know how the 4 reasons for non-compliance would be instantiated in these domains. The analysis shows that CPGs as well as tools promoting their implementation are necessary in clinical practice. So is their regular update to follow the evolution of the medical state of the art. However, recommendation evolution has to be validated by an editorial board prior to CDSS update.