Textual Guidelines versus Computable Guidelines: A Comparative Study in the Framework of the PRESGUID Project in Order to Appreciate the Impact of Guideline Format on Physician Compliance

Abstract

BACKGROUND

The compliance of physicians with the clinical practice guidelines (CPG) is insufficient and needs to be improved.

OBJECTIVE

To determine whether standalone computerized CPG within the PRESGUID project could improve compliance with the recommendations compared to the use of CPG in textual format.

METHOD

Comparative analyses of the responses made by two groups of resident physicians to a set of clinical cases. One group of residents had access to the CPG exclusively in textual format (paper document) while the second group had access to the CPG exclusively in computerized format within the PRESGUID software applications.

RESULTS

The computable CPG are more efficient than the paper-based CPG regarding responses in compliance with the recommendations especially those judged to be relevant by an expert.

CONCLUSION

These results should encourage the bodies responsible for diffusing CPG to promote the computable format and to facilitate the computerization process

INTRODUCTION

The Clinical Practice Guidelines are designed to improve practice as they provide a synthesis of the best and most up-to-date knowledge in relation with a specific clinical setting. They help guide decision-making towards optimal solutions for a given patient within this setting.

To ensure that this intention becomes a reality in everyday practice, measures need to be taken to improve physician compliance with the CPG recommendations [1].

Currently, CPG are distributed, for the most part, in textual format (paper-based or web-based publications in narrative documents). However, for several reasons, they have made little impact on daily practice [2]. It is crucial to seek full compliance with the guidelines, as partial application of CPG can be a source of medical errors [3].

Computerization of CPG can improve their level of use and their impact on clinical practice [4]. Several teams of researchers have been working in this direction [5]. The systems which have demonstrated their usefulness are essentially reminder systems coupled with the patient’s electronic medical record (EMR) which are most often implemented in innovative hospital information systems [6].

Integration of computable CPG into process-oriented information systems is an effective means of gradually injecting them into health-care processes and of improving their usefulness [7–9].

The PRESGUID project shares the same focus. It is being developed [10] to contribute to the improvement of health-care quality by providing CPG management tools which can be used in several health-care settings. In this study, a standalone version was used which is a forerunner of a future version being designed to improve integration into the clinical workflow.

This type of software application is of great interest in terms of usability but does it enhance compliance with CPG as compared with the paper-based version of these same CPG? This is the question which we endeavour to answer in this paper.

To this end, we undertook a study aimed at determining whether the use of computable CPG within PRESGUID software led to better compliance with the recommendations contained in the CPG than did the paper format.

MATERIAL AND METHODS

Our study was based on a comparative analysis of responses given by two groups of general medicine residents to a set of clinical cases related to pathologies dealt with in three CPG. One group of residents had access to the CPG only in textual format (paper-based document) whereas the second group had access only to the CPG in the computable format (PRESGUID web application).

CPG included and computerization method

We used the following CPG:

1. “Management of patients with essential high blood pressure (HBP) » (published by the ANAES -French National Agency for Accreditation and Evaluation in Health Care - April 2000) [11].

2. “Management strategies for the type 2 diabetic patient excluding complications management” (published by ANAES, March 2000) [11].

3. “The role of vasoactive agents in the management of peripheral arterial disease (PAD), stage II”. (published by the AMIS-2 group, April 2001) [12].

For each of these CPG, distributed in textual format by the bodies which produced them, we created a computerized version by using the production and distribution platform from the PRESGUID project [13]. The PRESGUID computable CPG are accessible via a dynamic web application allowing CPG consultation in guided mode. During PRESGUID CPG consultation, the user must capture the patient data required and inferred by the system, in order to produce guidelines suited to the patient profile. A complete description of the PRESGUID project has been published elsewhere [10].

Study protocol

Fifty-two residents were recruited randomly on a volunteer basis and were remunerated for participating in the study. None had previously taken part in the PRESGUID project. None had previously used this application.

A week before joining the study, and before being allocated to a group, all the residents received 10 minutes training in the use of the PRESGUID application and received a copy of the paper-based CPG that they were to study and keep.

The residents were later asked to attend three 50 minute sessions during which, on each occasion, they had to solve five clinical case questions. These sessions took place weekly. At the beginning of the first session, the residents were allocated at random to two groups of twenty-six. The residents in the “text-based CPG” group had to answer the clinical case questions consulting only the paper-based CPG. Those in the “Computable CPG” group used only the CPG implemented within PRESGUID. The clinical cases to be solved were identical in both groups. During all the sessions, the residents had to answer five questions on PAD, five questions on HBP and five questions on diabetes (NB : each session comprised a mix of questions requiring consultation of all three CPG).

To respond to these questions, the residents had to write down the management they advocated (diagnostic and therapeutic processes – with or without medication – and monitoring) for each of the situations described in the clinical cases by consulting, according to their group, either the computable CPG under PRESGUID or the paper-based CPG.

Data analysis plan

Following these sessions, all the responses were analysed in order to draw up an exhaustive list of the different « Management items » (MI) advocated by the residents. Management items consist of basic components in the decisions taken by these physicians using the CPG (computable or paper-based). The following are a number of examples: “Commence a low sodium diet”, “exploration by exercise electrocardiography”, “prescribe a statin”… The MI were checked by an expert physician operating blind (i.e. with no knowledge of the identity of the resident or of the group to which he/she belonged). They were then analysed using two separate approaches:

1. A quantitative approach: the expert checked each of the MI according to its compliance or non-compliance with the CPG guidelines. For each question j, we calculated the number n_i,j of compliant MI given by the resident i, that were divided by the total number T_j of possible compliant MI which could have been given for this question. So doing, a compliance rate CR_i,j was obtained using the formula shown in figure 1. The mean compliance rates were then calculated for each group of residents and for each CPG corresponding to the questions.

2. Qualitative approach: the MI were classified into six categories c of management (explorations, non drug-based treatment, drug-based treatment, treatment of comorbidities, patient education and follow-up). The MI quoted were distributed among these different categories. More precisely, for each question j, the expert determined the weight w_j,c of each of the above-mentioned categories in order to reflect their relative importance within the overall patient management (the sum of the weightings of the six categories always amounted to 100%). For each question j, each compliant MI_i,j,c indicated by the resident i was multiplied by the weighting w_j,c of the category of management to which it belonged. The sum of these MI was then divided by the total number T_j of compliant MI in order to define the weighted compliance rate (WCR, cf. figure 2). The mean weighted compliance rates were then calculated for each group of residents and for each CPG related to the questions.

Figure 1.

Compliance rate (CR) of resident i to question j

Figure 2.

Weighted Compliance Rate (WCR) of resident i to question j.

The aim of this subjective and expert-reliant category weighting was to provide an analysis of the scores obtained by the two groups taking into account the clinical relevance of certain MI as opposed to other less important items which were, nonetheless, recommended by the CPG. The main objective was to assess the level of compliance to the CPG recommendations which the medical expert esteemed to be most relevant in the clinical setting.

Statistical Methods

We compared the rates (CR and WCR) of the two groups using the Student test with correction for non-normality and after comparison of the variances using the Levene test. Tests were performed bilaterally with a significance threshold of 5%. SPSS v. 12.0.1 software was used to process the data.

RESULTS

Quantitative and qualitative analysis revealed statistically significant results in favor of better compliance to the guidelines by residents in the group using the PRESGUID computerized CPG as opposed to the group using the paper-based guidelines.

Quantitative analysis

The results given in table 1 show that the overall CR (i.e. all questions taken together) was significantly higher for the group using the computable CPG than for the group using the paper-based CPG.

Tableau 1.

CPG Compliance rate in the two groups of residents

Group	Text-based CPG	PRESGUID Comput. CPG	p
CPG
HBP	35.4 (sd 9.7 ; n=26)	43.6 (sd 9.9 ; n=26)	.004
Diabetes	24.2 (sd 8.9 ; n=26)	33.0 (sd 10.7 ; n=26)	.002
PAD	38.0 (sd 11.3 ; n=26)	48.9 (sd 11.2; n=14*)	.006
TOTAL	32.6 (sd 7.7)	39.3 (sd 9.7)	.008

^*Twelve of the twenty-six residents in the « computerized CPG » group used an erroneous computerized CPG for PAD. We omitted their responses regarding this CPG from the results analysis (number of residents n = 14 instead of 26 in the “computerized CPG” group for questions relative to PAD).

The CR per CPG (i.e. the questions dealing with a specific CPG) was also significantly higher for the “computerized CPG” group than for the “text-based CPG” group and whatever the CPG in question (HBP, Diabetes, PAD).

Qualitative analysis

The results of the qualitative analysis confirm the findings of the quantitative analysis and support the efficacy of the computerized CPG. Overall, or whatever the CPG, the weighted compliance rate was significantly higher in the group which had used the computerized CPG as opposed to the “text-based” group (cf. table 2).

Tableau 2.

Weighted CPG compliance rate in the two groups of residents

Group	Text-based CPG	PRESGUID Comput. CPG	p
CPG
HBP	46.7 (sd 13.0 ; n=26)	57.7 (sd 12.5 ; n=26)	.003
Diabetes	28.2 (sd 8.8 ; n=26)	39.7 (sd 12.3 ; n=26)	<.001
PAD	49.4 (sd 13.8 ; n=26)	59.0 (sd 13.3 ; n=14*)	.041
TOTAL	41.5 (sd 9.2)	49.5 (sd 11.1)	.006

^*cf. the remark on legend to table 1.

Thus, computerized CPG are shown to be more effective than text-based CPG regarding the production of responses in compliance with the guidelines and judged to be most relevant by the expert.

DISCUSSION

The aim of the study was to assess the impact of the CPG format on physician compliance. The results obtained, following a blind comparative study, show that compliance with the guidelines was significantly higher when the resident physicians used PRESGUID computerized CPG as opposed to paper-based CPG. This finding was validated for all the CPG studied.

The quantitative analysis, shows that the physicians using computable CPG, suggest a larger number of management items recommended by the CPG than the physicians using the paper-based version.

Qualitative analysis, based on clinical expertise, permits assessment of the impact of the CPG format on the essential or more or less secondary character of the recommendations adopted by the physicians in the decision-making process. The results obtained argue in favour of the greater clinical relevance of the management strategies proposed by the users of the PRESGUID computable CPG.

Nonetheless, this analysis is based upon the opinion of a single expert and thus raises the question of the reproducibility of the weightings attributed to the MI categories. These findings need to be confirmed, notably by drawing upon the consensual opinion of several experts.

In both the quantitative and qualitative analyses, we looked for compliance with the CPG recommendations. In the responses to the clinical cases questions, a non-compliant attitude can be conveyed both by the presence of MI not recommended in the CPG and by the absence of MI recommended in the guidelines. Moreover, as suggested in the work of Ramnarayan et al. [14], the non-recommended MI can be clinically non-significant or even dangerous or, in contrast, can be clinically significant and appropriate even if they are not mentioned in the CPG. The scoring system we used in our study did not attempt to distinguish and to analyse these different possibilities but to highlight the effective presence of MI recommended by the CPG in the responses supplied by the physicians in order to determine which medium (text-based CPG vs computable CPG) gave the best guideline ‘penetrance’.

There exists a host of factors which could possibly impact upon physician compliance with the CPG [15]. Among them, one could list: the nature and characteristics of the decision support system, the profile of the physicians using the system, the content and complexity of the CPG, the nature of the decision to be taken, etc…However, it would be mistaken to claim that the conditions in which our study was performed are representative of all the circumstances and contexts found in everyday practice.

Nevertheless, this study, conducted on trainee physicians, demonstrates some of the potential of computable CPG. It should be pointed out that the conditions of the study were particularly favorable in several respects to the group using the paper-based CPG. The residents in this group enjoyed optimal conditions far removed from real practice: the CPG had been studied prior to the test and were consulted during the decision-making process, an average of 15 minutes was allowed for the clinical case questions, etc. In contrast, the residents in the « computable CPG » group worked under sub-optimal conditions. PRESGUID is a standalone application which requires the patient’s (often numerous) characteristics to be filled in by hand – a time-consuming operation. In addition, the residents had received only a very brief training session and had not had access to the computable CPG other than during the clinical case sessions. Despite these handicaps, the PRESGUID application was shown to be superior to the conventional paper-based CPG format. One can be justified in believing that improvements such as the integration of this application into an EMR will enhance even further the efficiency of the system, as has been demonstrated with other computable CPG systems [16].

The development of CPG requires considerable time and human and financial resources. The current mode of diffusion, almost entirely limited to the paper format, has shown its limits [17] and raises the question of the profitability of such development processes which systematically focus on a text-based publication process, on paper or equivalent media. Implementation of the CPG using validated decision support systems should be pursued. Our study confirms this statement and can serve as a basis for further discussion and studies to ensure that such tools are made more widely available to health-care professionals and trainee physicians.

CONCLUSION

This paper focuses on the assessment of the contribution made by computable CPG to medical decision-making. By means of a study comparing computerized CPG versus paper-format CPG, it aims to determine whether computerization can lead to greater compliance with the reference, i.e. better implementation of clinical practice guidelines.

The findings in favour of the PRESGUID computerized system should encourage the bodies responsible for the diffusion of CPG to facilitate the computerization process and to promote their distribution in computable format.