Quality improvement in preoperative assessment by implementation of an electronic decision support tool

Abstract

Objectives

To evaluate the impact of the electronic decision support (eDS) tool ‘PReOPerative evaluation’ (PROP) on guideline adherence in preoperative assessment in statutory health care in Salzburg, Austria.

Materials and methods

The evaluation was designed as a non-randomized controlled trial with a historical control group (CG). In 2007, we consecutively recruited 1363 patients admitted for elective surgery, and evaluated the preoperative assessment. In 2008, PROP was implemented and available online. In 2009 we recruited 1148 patients preoperatively assessed using PROP (294 outpatients, 854 hospital sector). Our analysis includes full blood count, liver function tests, coagulation parameters, electrolytes, ECG, and chest x-ray.

Results

The number of tests/patient without indication was 3.39 in the CG vs 0.60 in the intervention group (IG) (p<0.001). 97.8% (CG) vs 31.5% (IG) received at least one unnecessary test. However, we also observed an increase in recommended tests not performed/patient (0.05±0.27 (CG) vs 0.55±1.00 (IG), p<0.001). 4.2% (CG) vs 30.1% (IG) missed at least one necessary test. The guideline adherence (correctly tested/not tested) improved distinctively for all tests (1.6% (CG) vs 49.3% (IG), p<0.001).

Discussion

PROP reduced the number of unnecessary tests/patient by 2.79 which implied a reduction of patients’ burden, and a relevant cut in unnecessary costs. However, the advantage in specificity caused an increase in the number of patients incorrectly not tested. Further research is required regarding the impact of PROP on perioperative outcomes.

Background and relevance

The usefulness of preoperative diagnostic assessment has been the subject of many studies. In 1985 a research group analyzed the usefulness of preoperative laboratory screening and concluded that in the absence of specific indications, routine preoperative laboratory tests contributed little to patient care and could reasonably be eliminated.¹ Another prospective cross-sectional study in healthy patients demonstrated that an average of 72.5% of preoperative tests ordered by surgeons, were considered not indicated by anesthesiologists.² According to a retrospective analysis at the Mayo Clinic, elective surgery in healthy patients without any prior testing will not lead to an increase in perioperative complications.³

A Health Technology Assessment (HTA) report in 1998 investigated the value of routine preoperative testing in healthy or asymptomatic adults. Tests very rarely led to changes in clinical management, and the predictive value of these tests regarding perioperative complications in asymptomatic patients was very low.⁴

In 2000 a large randomized controlled trial imposingly demonstrated that preoperative testing did not implicate any advantages compared to no-testing regarding perioperative adverse events in a cataract surgery population.⁵ Another study in 2004,⁶ again on cataract surgery, as well as a Cochrane review on the subject,⁷ showed similar results.

In 2001 it was pointed out that abnormal test results, occurring with a prevalence between 0.5% and 12% within the study population (n=544), showed no significant correlation with perioperative complications, American Society of Anesthesiologists (ASA) status, and severity of surgery in multivariate analysis.⁸ We came to a similar conclusion when analyzing the historical control group (CG) of this trial (1363 patients undergoing mixed types of elective surgical procedures in a secondary care hospital in Salzburg).⁹

In 2009 it was demonstrated in a randomized controlled pilot study (n=1061) that preoperative diagnostic testing did not reduce the rate of intra- and postoperative adverse events (risk ratio (RR) 1.0, and RR 0.8, respectively) compared to no testing in a mixed surgery population.¹⁰

Preoperative assessment is performed to keep the risk for intra- and postoperative adverse events to a minimum. Therefore, according to international guidelines and advisories, testing should be structured, selective, and based on medical history, physical examination, and type of surgery. None of these guidelines recommend routine tests.^11–14 In Austria, however, preoperative evaluation is usually performed according to local standards, which are frequently unstructured and non-selective. We were able to demonstrate that this approach leads to a substantial number of non-indicated tests.¹⁵

Based on this background, the project ‘PReOPerative evaluation’ (PROP) was initiated in the province of Salzburg, Austria in 2007 on behalf of the local health platform, which manages the state health funds and is intended to steer provision of care and financing across sectors.¹⁶ The aim of the project is the optimization of process quality by reducing unnecessary testing in preoperative assessment, thus decreasing diagnostic burden for patients, and reducing costs in the healthcare system. The project, initiated by G. Fritsch, is based on the guideline for preoperative evaluation released by the Austrian Society of Anaesthesiology, Resuscitation and Intensive Care Medicine (ÖGARI)¹⁷ which has been incorporated in an electronic decision support (eDS) tool (PROP) developed by S Klausner and K Entacher of the informatics department of Salzburg University of Applied Sciences. Since January 2008 the eDS tool has been available online (http://prop.fh-salzburg.ac.at) for all physicians working under contract with statutory public health insurance, and for all public and private hospitals in Salzburg. The tool prompts the physician to enter age, gender, medical history, and physical examination of the patient, and to categorize the severity of the planned surgical procedure by clicking the appropriate buttons via a user friendly interface. Using the programmed algorithm, PROP then displays a recommendation of tests on the screen, which is in accordance with the ÖGARI guideline.¹⁷ In a majority of cases no further testing is required. Age-related or routine testing is explicitly not recommended.

Objectives

The aim of this non-randomized controlled trial was to test the hypothesis that the implementation and usage of the eDS tool PROP would lead to a reduction of unnecessary preoperative testing, improve guideline adherence in preoperative assessment, and lead to a relevant cost reduction for the healthcare system. This evaluation study has been designed and carried out by AS, MF, and MH, who were not involved in the development or implementation of the eDS tool.

Materials and methods

The evaluation was designed as a non-randomized controlled trial with a historical CG and an intervention group (IG) using the eDS tool after its implementation. We obtained ethics approval for the study from the Ethics Committee of the Province of Salzburg (http://www.salzburg.gv.at/ethikkommission). All patient data were obtained and processed in a strictly pseudonymized way, complying with rules and regulations for the handling of health related personal data. From October through December 2007, before the implementation of PROP, we consecutively recruited 1363 patients planned for elective surgery in a public secondary care hospital (KH Schwarzach) in the province of Salzburg. We prospectively recorded the preoperative assessment of these patients regarding age, gender, medical history, physical examination, severity of the operation, and preoperative tests performed. Tests were performed either by general physicians (GPs) or in the hospital. We then analyzed the indication for the tests performed using the PROP algorithm based on the ÖGARI guideline, to identify indicated and unnecessary tests. Detailed methods and results of these analyses were published elsewhere.¹⁵ This cohort was used as the CG in our present study.

Following a testing phase, PROP was implemented and made available online mid-2008. After the implementation of PROP (April to July 2009) we consecutively recruited 854 patients planned for elective surgery whose preoperative assessment had been performed in a tertiary care hospital in Salzburg (Paracelsus Medical University) using PROP, and 294 patients planned for elective surgery whose perioperative assessment had been performed by a GP in the outpatient sector using PROP. Outpatient data were provided—anonymously—by the Salzburg Statutory Health Insurance. All tests performed in the outpatient cohort, from admittance to the hospital until the date of surgery (usually only 1 day), were included, using the database of the hospital. All data were pseudonymized at extraction. For the CG, we recorded age, gender, medical history, physical examination, severity of the operation, and all preoperative tests that had been performed. After matching these datasets with the PROP data entered by the physician using the PROP-ID, we analyzed the indication for performed tests using the PROP algorithm based on the ÖGARI guideline.¹⁷

We focused our analysis on six tests which we considered to be the most important and most frequently performed. These preoperative examinations included full blood count, liver function tests (ie, glutamic-pyruvic transaminase), coagulation parameters (ie, prothrombin time/INR), electrolytes (sodium and potassium), ECG, and chest x-ray.

All data were entered into an SPSS database and then analyzed by SPSS V.19.0. All analyses were performed with pseudonymized data. Using patient data and the algorithm of the PROP outcome, we determined the number of tests performed without indication for both IG and CG as well as the number of tests not performed in spite of recommendations of the guideline, and tested the difference using Welch tests. We then calculated the percentages of patients tested unnecessarily, of patients not tested though tests were recommended, and of patients assessed in complete accordance with the guideline, and compared the IG and CG using Fisher's exact test/χ² test.

Furthermore, average costs due to unnecessary tests per patient were calculated for the CG and for the IG, based on the reimbursement rates of Salzburg Statutory Health Insurance for the respective tests. From the difference in costs and the number of elective surgical procedures performed in Austria per year, we calculated a rough estimate of possible savings for the six most commonly applied preoperative tests.

Results

Study population

Descriptive data of the CG (n=1363) and IG (n=1148) regarding age, sex, severity of surgery, and ASA status are shown in table 1.

Table 1.

Descriptive data of the control group and intervention group

	Control group (n=1363)	Intervention group (n=1148)	p Value
Age (years mean±SD)	50.2±19.9	51.26±18.7	0.205*
Sex (% male)	43.9	39.6	0.031†
Severity of surgery: minor vs major (% minor surgery‡)	83.7	83.1	0.665†
ASA status ≥3 (% of all)	5.1	2.8	<0.05†

*Independent t test.

†χ² test (Fisher's exact test).

‡Definition of minor surgery (all of the following): duration of operation <2 h; expected loss of blood <500 ml; no opening of visceral cavity (except in case of diagnostic laparoscopy, endoscopic cholecystectomy, and appendectomy). Definition of major surgery (any of the following): duration of operation ≥2 h; expected loss of blood ≥500 ml); opening of visceral cavity; potential massive respiratory or hemodynamic effects due to operation.

Comparison of guideline adherence between CG and IG

The total number of unnecessary tests per patient decreased significantly from 3.39±1.44 in the CG to 0.60±1.06 after the implementation of PROP (p<0.001), as did the percentage of patients unnecessarily receiving any specific test. In total, in the CG, 97.8% of all patients received at least one unnecessary test. This percentage decreased to 31.5% of all patients after the implementation of PROP (31.0% in the outpatient and 31.7% in the hospital sector). In the average of the six tests analyzed in our study, 9.9% of all intervention patients still received a test deemed unnecessary by the guideline (maximum 15.2% (liver function tests), minimum 1.9% (chest x-ray)).

However, we also observed an increase in patients incorrectly not tested. While in the CG only 0.05±0.27 tests recommended by the guideline per patient were not performed, in the IG 0.55±1.00 recommended tests/patient were omitted (p<0.001). At least one necessary test was not performed in 4.2% of the control patients versus 30.1% of the intervention patients. In the average of the six tests analyzed in our study, 9.2% of all intervention patients had a necessary test omitted (maximum 14.8% (full blood count), minimum 1.0% (liver function tests)).Table 2 shows the percentages of patients who needed a particular test and got it (‘true positives’), needed it and did not get it (‘false negatives’), did not need it, but got it (‘false positives’), and did not need it and did not get it (‘true negatives’).

Table 2.

Percentages of patients correctly tested, correctly not tested, falsely tested, and falsely not tested before and after the implementation of PROP

	Before PROP (n=1363)		With PROP (n=1148)
	Test indicated	Test not indicated	Test indicated	Test not indicated
Electrolytes
Test performed	24.7	55.6	22.1	11.5*
Test not performed	1.6	18.1	10.8*	55.6
Coagulation tests
Test performed	12.1	81.7	11.1	12.5*
Test not performed	0.5	5.7	10.6*	65.8
Full blood count
Test performed	24.7	72.7	25.0	11.9*
Test not performed	0.5	2.1	14.8*	48.3
Liver function tests
Test performed	1.1	58.2	1.5	15.2*
Test not performed	0.1	40.6	1.0**	82.3
ECG
Test performed	8.6	45.3	8.6	6.7*
Test not performed	0.6	45.5	9.0*	75.7
Chest x-ray
Test performed	4.9	25.2	3.5	1.9*
Test not performed	1.9	68.0	9.3*	85.3

*p<0.001 compared to control, χ² test.

**p>0.05 compared to control, χ² test.

PROP, PReOPerative evaluation.

Overall guideline adherence (including correctly tested as well as correctly not tested patients) increased from 1.6% of all patients in the CG to 49.3% of all patients in the IG (p<0.001, 50.0% in the outpatient sector and 49.1% in the hospital sector). The results regarding guideline adherence separately for all six tests are shown in table 3.

Table 3.

Percentages of patients preoperatively tested guideline adherent (all patients ‘tested’ and ‘not tested’ in accordance with guideline recommendations)

	Control group (n=1363)	Intervention group (n=1148)	p Value*
Electrolytes	42.8	77.7	<0.001
Coagulation parameters	17.8	76.8	<0.001
Full blood count	26.7	73.3	<0.001
Liver function tests	41.7	83.8	<0.001
Electrocardiogram	54.1	84.3	<0.001
Chest x-ray	72.9	88.8	<0.001
Total (all 6 tests guideline adherent)	1.6	49.3	<0.001

*Fisher's exact test.

Estimation of possible cost savings by the implementation of PROP

Average expenditure for unnecessary tests using the reimbursement rates of Salzburg Statutory Health Insurance are shown in table 4. While in the CG €40.48 are spent per patient for unnecessary preoperative tests, this expenditure comes down to €5.82 per patient when PROP is used, yielding cost savings of €34.66 per patient with only the most frequent tests being taken into account. If the costs of €29 incentive for the GPs to use PROP are deducted, there remains a net saving of €6/patient. This assumption represents the most conservative way of estimating possible savings, as the incentive is only disbursed to physicians in the outpatient sector. Official statistics are estimating an annual rate of 1.16 million surgical procedures performed in Austria, of which about 75% (870 000) are elective. Multiplying the number of elective surgical procedures with the PROP associated cost savings per patient yields possible savings of about €5 million/year for the Austrian health system by a nationwide implementation of PROP for only the six tests studied in our trial. Using the same rates and tests, routine testing accounts for approximately €35 million total costs for preoperative diagnostics in elective surgery in Austria per year for only the six tests used in our study. This yields savings of about 15% by implementing PROP. Although these savings appear comparatively small in relation to the total health expenses of about €30 billion yearly (Statistics Austria), they must be considered substantial in relation to the total expenditure currently spent for preoperative diagnostics.

Table 4.

Health expenditure for unnecessary preoperative testing before and after the implementation of PROP

Test	Costs per test (€*)	Control group		Intervention group
		Unnecessary tests per patient	Unnecessary costs per patient (€)	Unnecessary tests per patient	Unnecessary costs per patient (€)
Electrolytes (sodium, potassium)	3.20	0.56	1.80	0.11	0.36
Coagulation parameters (prothrombin time, INR)	13.30	0.82	10.90	0.13	1.72
Full blood count (hemoglobin, leucocytes, thrombocytes)	6.97	0.73	5.09	0.12	0.84
Liver function tests (GOT, GPT)	3.20	0.58	1.86	0.15	0.48
ECG	22.20	0.45	9.99	0.07	1.55
Chest x-ray	43.35	0.25	10.84	0.02	0.87
Sum			40.48		5.82

*Rates of the Salzburg Statutory Health Insurance.

GOT, glutamic oxaloacetic transaminase; GPT, glutamic pyruvic transaminase; PROP, PReOPerative evaluation.

Discussion

Our study showed a highly significant reduction in unnecessary testing and an improvement in overall guideline adherence by implementation of the eDS tool PROP. PROP reduced the number of unnecessary tests per patient by 2.79, corresponding to an absolute reduction of 82%. This improvement is exclusively due to the omission of ‘routine’ testing, which is still a very common practice in Austria. The decrease in the number of unnecessary (not indicated) tests not only lowers the burden of diagnostic testing for the patient but also leads to a substantial reduction of healthcare expenditure in preoperative assessment.

Guidelines to reduce unnecessary perioperative testing have been published for many years, but guideline implementation is still a problem that needs to be solved. The implementation of eDS tools, like PROP, may help to improve guideline adherence. This observation is in accordance with other experiences with eDS tools that are beginning to be implemented for various quality improvement purposes throughout the medical field. Especially in medication safety and improvement of proper prescribing, eDS has been implemented and tested.¹⁸ The effects seen in these therapeutic studies are far less than the improvement in guideline adherence we observed in our study. This may in part be due to a reimbursement incentive given to the GPs when using PROP in our project. On the other hand, no incentives were given to the hospitals where improvement has been comparably good.

While the elimination of unnecessary tests is a wanted result of the implementation of PROP, the use of the tool also led to an increase in the omission of tests recommended by the guideline despite the fact that the tool explicitly asked for all these tests. There are several possible explanations for this phenomenon. In the intervention PROP was tested against routine testing which obviously includes all tests, thus automatically avoiding the omission of necessary tests. In the IG physicians were led from routine testing to reflective testing using the guideline. Furthermore, the rather mechanistic instruction of the guideline may lead physicians to deliberately decide whether a particular test is needed for a particular patient or not. Physicians may then deliberately deviate from the guideline for patient-related reasons (eg, the test has recently been performed and revealed a normal result or the test is not deemed necessary by the physician because the (chronic) disease the test is looking for is already known, and the test would not add any additional information). This deliberate deviation from the guideline applies to both the performance of tests deemed unnecessary by the guideline as well as the omission of tests deemed necessary by the guideline. Indeed the percentages of patients with omitted tests are quite similar to the percentages of patients receiving unnecessary tests in the IG of our study (both approximately 10%). Further research is needed to address the particular reasons for deviation from the guideline.

From our data we estimated cost savings of about €5 million for only the most common tests applied in preoperative diagnostics, using the most inexpensive rates of statutory public health insurance. Depending on the calculation base for costs we estimate that savings (including additional tests apart from the most common ones included in our study) could be up to €30 million per year for Austria. Our estimations on cost savings are supported by previous similar investigations,^{15 19} but need to be verified by further economic evaluation research with detailed cost analyses in the outpatient and the hospital sector.

Also, possible cost effects of the omission of recommended tests must be considered. The omission of tests will lead to savings regarding the cost of the tests no matter whether the tests were recommended by the guideline or not. Thus the deliberate omission of a recommended test by a reflective physician (eg, because the test had recently been performed) will lead to further savings despite the inherent deviation from the guideline. On the other hand it could be hypothesized that the omission of recommended tests may increase perioperative risk and thus cause additional costs for prolonged hospitalization or the treatment of possible complications. Looking at our previous study which showed no measurable correlation between perioperative outcome and pathologic preoperative tests, this seems very unlikely.⁹ The study of Chung et al¹⁰ also showed quite clearly that the omission of routine testing is not associated with an increase in risk or complications. Nonetheless, a randomized controlled outcome study of PROP would be necessary to validly settle this issue.

Recently a number of papers have been published and summarized in an editorial that specifically addresses cost effectiveness of IT interventions. Especially for clinical decision support tools there appears to be a wide variation in effect, and our knowledge about effects and how they are modulated is still quite limited.²⁰

The process of successfully implementing a clinical practice guideline to change day-to-day practice is rather challenging. An investigation in a Canadian hospital pointed out that non-compliance with published guidelines regarding preoperative assessment varies widely (5–98%).²¹ This implies that the strategy used to implement a clinical guideline may play a crucial role.²² A new guideline implemented via memorandum sent to all physicians in a hospital led to a significant reduction in total tests after 2 years, but only in 15% of cases was the guideline followed exactly.²³ Greer et al²⁴ showed only a slight reduction in unnecessary testing within a period of 6 months using information, for example lectures and printed material, as implementation strategy.

Sufficient strategies are essential to translate guidelines into practice. The implementation of a guideline via a user-friendly eDS tool, that can be integrated in the practice software in a physician's office or in the hospital, seems most promising according to our results. Unfortunately, so far PROP could not be integrated into the electronic health records used in the outpatient sector in Austria, because there are more than a dozen different systems in use with limited compatibility. This certainly represents a considerable drawback for nationwide implementation, and may partially explain guideline deviation in addition to the mechanisms described above. Despite current incompatibility with electronic health records, PROP as a standalone was quite successful in improving overall guideline adherence. Inclusion in the electronic health records may further improve the implementation of PROP and is a goal currently pursued by health policy makers in Austria.

The success of PROP in our study may largely have been due to incentives for the physicians in the outpatient sector: the GP receives a remuneration of €29 per patient for using PROP, and preoperative diagnostic tests are only reimbursed by statutory public health insurance if they conform to guideline recommendations to counteract unstructured screening. Of course physicians may justify additional tests by providing a specific indication if they feel the test is needed.

Since the project showed impressive improvements in guideline adherence as well as economic benefits, the Austrian Ministry of Health decided to implement a nationwide regulation for preoperative assessment using the ÖGARI guideline, which also is the basis of PROP. A task force consisting of all medical disciplines was installed to agree on the guideline contents and to plan adaptations of the decision support tool to enable a nationwide roll-out and inclusion in electronic health records. Currently, the eDS tool PROP is being rolled out in Austria as a national program but still awaits integration in standardized electronic health records.²⁵

Limitations

Results derived from controlled studies with historical controls are known to overestimate effects due to unaccounted differences between the groups and progress in care not related to the intervention tested. The historical CG in our study included a mixed selection of surgery types and patients may represent a slightly different setting, and thus may not be entirely comparable to the IG. We used a historical CG because a randomized controlled comparison was not feasible due to objection from statutory health insurance and lack of resources. However, the extent of quality improvement due to the implementation of the eDS tool is so large that it can certainly not be attributed to other accidental differences between the groups. Even if we take into account that effects are usually overestimated in the comparison with historical controls, we can be quite certain that true and extensive quality improvement has been achieved.

Another shortcoming of our study is the fact that we could not obtain outcome data for the IG, and thus we cannot directly conclude from our results that the reduction of testing deemed unnecessary by the guideline does not lead to an increase in peri- and postoperative adverse events. Looking at the literature this seems very unlikely. Also, our data do not allow any final conclusion on a possible detrimental effect on outcome by omitting tests that would have been recommended by the guideline. Although the number of ‘false negatives’ (ie, tests that were not performed though recommended) was small, this issue raises crucial concerns. It is known from other studies that the introduction of electronic tools not only reduces error but also in itself leads to error, especially in tools addressing medication safety. The phenomenon is referred to as e-iatrogenesis, and may—among the other possible causes discussed above—also play a role in still ameliorable guideline adherence despite correct recommendations by PROP.²⁶ The exact reasons for the omission of recommended tests cannot be inferred from our data, and outcome research is strongly recommended to exclude possible negative effects of omitted tests on perioperative events.

Conclusion

The implementation of the eDS tool PROP leads to a substantial improvement in guideline adherence compared to the historical CG. This improvement is accompanied by a reduction in patient burden and costs. A small increase in the omission of tests that are recommended by the guideline can also be observed and raises some concern. Further research is required to investigate the effect of PROP on peri- and postoperative outcome.