Consequences of the Verification of Completeness in Clinical Practice Guideline Modeling: a Theoretical and Empirical Study with Hypertension

Abstract

Building clinical decision support systems requires a formalization of clinical practice guidelines (CPGs) including the verification of completeness to ensure all medically relevant situations are addressed. Recommendations that rely on completed knowledge cannot be but expert-based. Using French hypertension management guidelines, we characterized the status of a patient profile as evidence-based (EB), consensus-based (CB), or expert-based (XB). The distribution of these status on the formal patient profiles of ASTIGM knowledge base showed that 12.6% (0.5% EB and 12.1% CB) lead to explicit CPG recommendations. The same analysis on a sample of 435 actual patients medical records showed that 55% were covered by CPGs. The characterization of guideline-based CDSSs should be based on empirical data estimated from the target population of CPGs.

INTRODUCTION

In order to provide optimal care, physicians are expected to follow evidence-based medicine (EBM) and make explicit, judicious, and conscientious use of current best evidence from health care research in decisions about the care of individuals and populations. Usually developed by health professional societies or national health agencies, clinical practice guidelines (CPGs) are intended to be information resources synthesizing evidence-based recommendations. They are classically presented as therapeutic recommendations given with grades of evidence for a set of theoretical clinical situations. Evidence has been pre-graded for validity from health care research by people with expertise in research methods. CPGs are thus expected to reduce inappropriate variations and improve the quality of clinical care and patient outcomes. Yet, evidence is often missing, e.g. over half of the recommendations of the Yale Guideline Recommendation corpus did not indicate strength of evidence,² and the majority of guidelines actually come from value judgment based on organizational preferences regarding the various risks and benefits for a given population of a medical intervention. In addition, when evidence is available, only a limited proportion comes from randomized controlled trials, the gold standard of scientific evidence, e.g. only 11% of ACC/AHA CPGs are classified as level of evidence A.¹

Numerous CPGs have been developed in narrative format and disseminated as paper-based and electronic documents. But many studies have shown that the sole dissemination of CPGs has nearly no impact on physician behavior. On the opposite, several reviews³ suggest that clinical decision support systems (CDSSs) are appropriate tools to promote CPG use. Defined as any software in which characteristics of individual patients are matched to a computerized knowledge base (KB) for the purpose of generating patient-specific assessments or recommendations, CDSSs rely on computerized versions of original textual CPGs. Despite the development of numerous guideline representation formalisms, the translation of textual CPGs into computerized KBs is still difficult and expensive.⁴ Shiffman et al⁵ have proposed a multi-step process to systematize and make explicit the translation of document-based knowledge into CDSSs including atomization, deabstraction, disambiguation and verification of completeness. The verification of completeness ensures that all medically relevant combinations of condition states are anticipated and addressed by the KB. This is an important step since a CDSS has to be comprehensive to minimize the risk of clinical practice variations and medical errors.

ASTI⁶ is a prototype guideline-based CDSS applied to therapeutic prescribing in primary care. Among other pathologies, ASTI is currently applied to the management of essential arterial hypertension (AHT) based on the French AHT guidelines.⁷ The “guiding mode” of ASTI (ASTI-GM) has been developed to provide guidance in all situations a general practitioner (GP) is likely to face. From the modeling process of CPGs, we defined three exclusive categories of patient profiles according to the strength of recommendations provided in CPGs. First, evidence-based, or EB, profiles are patient profiles for which there exist EBM recommendations with the explicit mention of a grade, either A, B, or C, in CPGs. Then, consensus-based, or CB, profiles are patient profiles for which, in the absence of proven literature, the panel of specialists in charge of CPG development considered there is a consensus to recommend therapeutic options explicitly stated in the CPGs document. Finally, expert-based, or XB, profiles are the other profiles, not explicitly stated in CPGs, that correspond to the “gaps” of CPGs, for which therapeutic options are expert-based.

The aim of this exploratory work is to study the consequences of the verification of completeness step in the knowledge modeling of ASTI-GM. From a theoretical point of view, we have evaluated the number of therapeutic propositions added by completion, thus non strictly guideline-based, and studied the theoretical distribution of formal patient profiles represented in the KB according to the three categories EB, CB or XB. A second objective was to estimate the empirical distribution computed on actual patient profiles from a data set built from electronic medical records (EMR) of a Parisian medical practice (France).

MATERIAL

Recommendations of French AHT CPGs

French AHT recommendations consist of declarative statements that associate a particular therapy to some clinical comorbidities, namely elderly, antecedent of stroke, heart disease including stable angina, myocardial infarction (MI), or ischemic heart failure, diabetes and renal disease. Drugs have to be chosen among five therapeutic classes: beta-blockers (BBs), ACE inhibitors (ACEi), angiotensin II receptor blockers (ARBs), calcium channel blockers (CCBs), and diuretics (Ds). The general therapeutic strategy is to start treatment with a monotherapy T₁, then add a new drug T₂ and switch to bitherapy T₁ + T₂, if T₁ is not efficient to normalize blood pressure, or substitute T₁ in case it is not tolerated. The same principles apply to go from bitherapy to tritherapy. However, in French AHT CPGs, recommendations are given with a grade of evidence, and are thus evidence-based, only for the initiation of the pharmacological treatment of patients with only one comorbidity. For instance, “thiazide diuretics (TDs) or long-acting dihydropyridine CCBs are recommended for initial therapy of elderly (grade A)”. Once the initial monotherapy is given, following therapeutic steps can be built on the basis of drug potentiation principles which are described, as general principles, in the CPGs document: from any of the 5 recommended drug classes, synergistic bi and tritherapies are proposed but no strength of evidence is provided. For instance, TDs and CCBs are preferably combined together or with BBs, ACEi, and ARBs. As a consequence, the recommended therapeutic sequence for elderly is (TDs or CCBs; TDs + CCBs; TDs + CCBs+(BBs or ACEi or ARBs)). Clinical conditions combining at least two comorbidities (e.g. elderly and diabetes) are not even mentioned in CPGs: there is no drug therapy proposition whether it concerns the initial monotherapy or the drugs to combine in bi or tritherapy. Non-tolerance or contraindications to recommended drugs is poorly handled by CPGs: in a few patient profiles, it is recommended with and without grade of evidence, depending on patient profiles, that non-tolerated ACEi could be replaced by ARBs. However, contraindications to other recommended drugs are not explored.

Modeling framework of textual CPGs

We formalized textual CPGs knowledge as a two-level decision tree made of a clinical level to represent clinical patient profiles and a therapeutic level to explore strategies that should be recommended from the analysis of the patient therapeutic history.

To build the clinical level, we first proceeded to the atomization step to extract single concepts (stable angina, diabetes, renal disease, etc.). Then, we carried on the deabstraction step to adjust the level of generality at which decision variables or actions were described in CPGs to permit the operationalization in a clinical setting (renal disease has been specified as glomerular filtration rate < 60 ml/min). The disambiguation step tried to establish a single semantic interpretation for the recommendation statement. The most important step is the verification of completeness to insure that implicit modalities of identified criteria have been explicited. The principle is to check at each node of the decision tree that modalities of corresponding criteria are exhaustive and mutually exclusive. The paths resulting from the expansion of the clinical level of the decision tree represent by construction a noso-logical repository of all possible clinical situations.

The therapeutic level is developed for each clinical situation elicited at the clinical level to make patient-specific according to her therapeutic history a clinical-condition-specific therapy. We handled the management of both therapeutic efficiency and non-tolerance. The first step is to identify the therapeutic sequences limited to tritherapies, (T₁; T₁ + T₂; T₁ + T₂ + T₃), adapted to each theoretical clinical situation. Because of CPGs incompleteness, most initial monotherapies to be recommended are missing. For clinical profiles explicitly described and associated to graded recommendations in CPGs (only one comorbidity), T₁ and sometimes T₂ are given. For clinical profiles existing in CPGs but without graded recommendation, T₁ is given but without evidence. In both cases, as T₁ is given then the whole therapeutic sequence (T₁; T₁+T₂; T₁+T₂+ T₃) could be built from drug combination principles but without evidence. For clinical profiles not covered by CPGs, thus built by the verification of completeness step, T₁ is unknown as well as the appropriate therapeutic sequence. All the gaps were filled in by an expert that participated to the development of AHT CPGs. To make ASTI-GM propositions more comprehensive, we also completed the KB to handle non-tolerance and added alternative therapeutic options to take into account drug substitutes and manage patients with contraindications to recommended drugs.

Data set of actual patient profiles

In March 2007, an anonymized data set has been built from the EMR system of a urban medical practice, located in Paris, France, and involving 5 GPs. Among more than 15,000 records, 669 patients with hypertension were selected using the presence of the keyword “AHT” in the antecedent section of medical records. For each selected patient with confirmed AHT, drug orders including at least one antihypertensive drug, posterior to December 2005, date of CPGs publication, and corresponding to a therapeutic decision (prescription of a new treatment or modification of the current one) were analyzed. For each order, patient data were collected according to the set of 24 criteria used in ASTI-GM KB. Such patient profiles include clinical data (comorbidities), current therapy level, and drug contraindications. The data set is made of 435 records (orders and their associated patient profiles).

METHOD

The paths resulting from the entire expansion of the two-level decision tree form an exclusive set of 44,571 atomic KB recommendations. A therapeutic solution S, either evidence-based, or recommended with no evidence in CPGs, or expert-based, is attached to any formal patient profile P. Each KB recommendation is represented by a rule, noted R = (P, S).

Status: from recommendations to profiles

The status of a recommendation R of the KB can be among 3 exclusive values, (i) evidence-based, or EB, when R exactly matches an explicit EBM statement in CPGs, (ii) consensus-based, or CB, when R matches a statement explicitly provided in CPGs but without evidence, and (iii) expert-based, or XB, when R is not explicitly stated in CPGs. We state that the status of a recommendation R of the KB is determined by its patient profile P. By extension, the status of a profile P is the status of the recommendation R it is eligible for. As a result, we consider 3 exclusive categories of patient profiles, EB, CB or XB.

Dimensions of status determination

From the modeling of French AHT CPGs, we identified 3 dimensions characterizing the status of a profile P which pertain to the patient’s clinical condition, her current treatment, and contraindicated drugs.

A. Clinical condition

We state that P supports EB recommendations from the clinical point of view, noted EB_cl(P), if P corresponds to a situation for which there exist EBM recommendations. As previously mentioned, this is true when either no comorbidity is associated to hypertension, or there is at most one comorbidity among: elderly, antecedent of stroke, MI, stable angina, ischemic cardiac failure, (diabetes ∧ microalbuminury). We state that P supports CB recommendations from the clinical point of view, noted CB_cl(P), if P corresponds to a situation for which CB recommendations exist. In French AHT CPGs, CB_cl(P) is true in 4 contexts: renal disease alone, left ventricle hypertrophy alone, (renal disease ∧ diabetes), (MI ∧ ischemic cardiac failure). P supports XB recommendations from the clinical point of view, noted XB_cl(P), if EB_cl(P) and CB_cl(P) are false. This is the case for profiles generated by the completion, which correspond to multiple comorbidities, e.g. AHT with MI and diabetes and elderly.

B. Level of drug association

As EBM recommendations exist in the case of drug therapy initiation, we state that P supports EB recommendations from the therapy point of view, noted EB_th(P), when P includes (therapy level = 0). Preferred drug combinations are recommended up to tritherapies on a consensus basis. Therefore, P supports CB recommendations from the therapy point of view, noted CB_th(P) when the current treatment is at most a bitherapy (thus may be switched to a still recommended tritherapy). In all other situations, P supports XB recommendations from the therapy point of view, noted XB_th(P).

C. Contraindications and non-tolerance

P supports EB recommendations from the contraindication point of view, noted EB_ci(P), when there is no contraindication to recommended drugs, except for ACE_i in the particular case of ischemic cardiac failure where a graded recommendation is provided in CPGs. Except in this case, contraindications to ACE_i are handled by CPGs with no grade of evidence, recommending substitution by ARBs. As a result, P supports CB recommendations from the contraindication point of view, noted CB_ci(P), when ACE_i are contraindicated. Other contraindications are not specifically considered in CPGs. Thus, P supports XB recommendations from the contraindication point of view, noted XB_ci(P), in the case of other drug contraindications.

Determination of the status of patient profiles

Let P be any patient profile, the status of P are defined as follows:

• EB profile: P is EB if it is EB on the clinical side, EB on the therapy side, and EB on the contraindication side.

• CB profile: P is CB if it is not fully EB, but is clinically CB or EB, therapeutically CB or EB, and CB or EB with respect to contraindications.

• XB profile: By construction, a patient profile is XB, if it is neither EB nor CB.

Experiments

We have developed an algorithm based on the above specifications to yield the status of any patient profile with respect to French AHT CPGs. It has been first applied to ASTI-GM KB to yield the status distribution in EB, CB and XB categories of formal patients handled by the KB. Then, it was applied to the data set to yield an empirical distribution of the status among actual hypertensive patients.

RESULTS

Theoretical distribution

Table 1 reports the status distribution of the 44,571 formal patient profiles of the expanded KB of ASTI-GM.

Table 1:

Distribution of formal KB patient profiles.

Profile types	n	Percentage
EB profiles	206	0.5%
CB profiles	5,424	12.1%
XB profiles	38,941	87.4%
Total	44,571	100.0%

Figure 1 illustrates the cumulative impact on profile status categorization accounting for the clinical (first bar), the therapy (second bar), and contraindications (third bar) dimensions to yield the final theoretical distribution among EB, CB, and XB profiles in the KB.

Figure 1:

Progressive construction of status profile distribution with formal patient profiles (left to right).

Empirical distribution

The same classification has been done on the 435 actual patient profiles of the data set (Tab. 2).

Table 2:

Distribution of actual patient profiles.

Profile types	n	Percentage
EB profiles	36	8.3%
CB profiles	204	46.9%
XB profiles	195	44.8%
Total	435	100.0%

Figure 2 illustrates, on actual patient profiles, the cumulative impact on status of the 3 dimensions that lead to the final status distribution.

Figure 2:

Progressive construction of status profile distribution with actual patients (left to right).

DISCUSSION

Formal KB profiles define the “theoretical” scope of ASTI-GM i.e. the set of all patient profiles that could possibly be faced by GPs. Results show that only 12.6% (Tab. 1) are explicitly covered by CPG recommendations, among which only 0.5% are EB. The analysis of the progressive categorization of formal KB profiles (Fig. 1) shows that the verification of completeness is concentrated at the clinical level of the decision tree and generates 72.3% of XB_cl profiles. This indicates that most added formal profiles concern the explicitation of complex clinical conditions (more than one comorbidity in addition to AHT). Completion according to therapy and contraindications dimensions only adds few profiles (+15.1%).

Since 87.4% of KB recommendations are expert-based one can wonder whether ASTI-GM should be considered as an expert-system or a guideline-based CDSS. Although anticipated, some KB profiles are never, or exceptionally, encountered in practice whereas some about the few EB profiles might be frequent. Thus, a solely formal perspective that does not integrate occurrence frequency of actual patient profiles is not enough to characterize the practical scope of the KB. The data set provides such an empirical distribution on actual patients. Figure 2 shows that, at the clinical level, the majority of actual patients are under the scope of EB_cl profiles, whereas “only” 24.4% correspond to XB_cl profiles. Considering the therapy and contraindication dimensions generates a high decrease of EB profiles, from 68.7% to 8.3%. Finally, since 46.9% of the data set correspond to CB profiles, 55.2% of actual patients are under the scope of explicit CPGs recommendations. Thus, about one half of actual patients remain in XB profiles mainly because they are treated with multiple drugs. This suggests that the difficulty of AHT management as a chronic disease might not proceed from the clinical characteristics of patients but rather from the complexity of drug management on the long term for which evidence is missing. For XB profiles, there is no guideline-based recommendations. CDSSs could fill CPG gaps, but without official agreements on how to fill these gaps, GPs would be less likely to follow such expert-based recommendations.

Limitations of this study include the limited domain (AHT management), the choice of specific French CPGs and the analysis of only one general practice. In addition, as ASTI-GM is a prototype guideline-based CDSS, it is not actually used by GPs and the data set has been retrospectively built from poorly coded medical records. The selection process may thus not ensure the representativeness of the studied patient sample.

CONCLUSION

The systematic formalization of CPGs through knowledge completion extends the theoretical coverage of the KB to integrate formal situations not explicitly mentioned in CPGs. Simply counting the proportion of EB, CB and XB rules in the KB does not allow to classify the CDSS as guideline-based or expert-based since some rules may be used more frequently than others. The confrontation to actual patient data illustrates that despite poor theoretical CPG coverage, actual CPG coverage of a CDSS might be substantial. The practical experiment we conducted to estimate the empirical distribution of rules permitted to measure that ASTI-GM would provide guideline-based recommendations for 55% of the patients. The characterization of the coverage of guideline-based CDSSs should take into account their target population.