Morbidity and mortality audits

Abstract

PROBLEM BEING ADDRESSED

While professions hold their members responsible for self-regulation, many physicians have insufficient information about outcome measures in their practices to judge performance and are inexperienced in performing audits to gather the information they need to judge performance.

OBJECTIVE OF PROGRAM

To develop a structure and process to support family doctors with little experience in doing quality improvement studies to conduct morbidity and mortality (M&M) audits.

PROGRAM DESCRIPTION

A family medicine teaching group provides members on a rotating basis to an M&M review committee. The committee meets eight times a year and has done four audits, the most comprehensive on the topic of preventable hospital admissions. Both implicit and explicit criteria were incorporated into decision making. Strengths and limitations of the audit process and practice changes that resulted from the audit are discussed.

CONCLUSION

Morbidity and mortality audits can vary in rigour. To promote physicians’ interest in and commitment to audits, factors considered should reflect the goals, needs, skills, and time available of the physicians involved. Practical learning often results from simple projects.

EDITOR’S KEY POINTS.

• This article presents a quality improvement program assessing medical practice (morbidity and mortality outcomes) in a family medicine centre in a university teaching hospital in Montreal, Que.

• Criteria for structuring the program are described. Although quality evaluation methods are becoming more and more sophisticated, a less structured process seemed more useful in an ambulatory setting.

• The authors present an example of an audit on the preventable causes of hospitalization in their centre.

POINTS DE REPÈRE DU RÉDACTEUR.

• Cet article présente le programme d’évaluation de la qualité de l’exercice professionnel (mortalité et morbidité) implanté dans le centre de médecine familiale d’un hôpital universitaire montréalais.

• Les conditions nécessaires pour structurer ce programme sont décrites. Bien que l’évaluation de la qualité de l’exercice professionnel soit de plus en plus sophistiquée, un processus moins structuré semble plus utile dans un milieu ambulatoire.

• Les auteurs présentent un exemple d’évaluation portant sur les causes évitables d’hospitalisation dans leur milieu.

Medical practice within the formal structure of the profession must be self-regulated.^1,2 The quality of patient care, clinical support services, organizational functions, and if necessary, programs of improvement, must be assessed.³ Clinical audits of quality of care are systematic critical analyses of procedures used for diagnosis, treatment, resource use, outcomes, and quality of life.⁴ Audits are more common in specialty disciplines than in family practice.^5,6

Reported benefits from British general practice audits include improved collegial communications, better patient care and practice administration, and increased professional satisfaction.⁷ A national system of Medical Audit Advisory Groups in the United Kingdom, which facilitate general practice audits, has, however, had varying success.^8-11 The Royal Australian College of General Practitioners promotes quality assessment as part of physicians’ maintenance of competence program.¹² Doctors who have been involved in these audits indicate that they teach about real, rather than perceived, practice.¹³ In Canada, little has been published on family practice audit experiences.¹⁴

A morbidity and mortality (M&M) review is a specific audit that targets negative conditions or outcomes.⁵ In the United Kingdom, these reviews are reported to be of value because there is often insufficient information on patients’ deaths.¹⁵ An abnormal audit finding could be an isolated event, but it might prompt further exploration into practice patterns. For example, an adverse drug outcome might prompt auditors to conduct a systematic audit of a predetermined number of cases in which that drug had been used. This more structured and rate-based process might be influenced by ease of data retrieval from files, problem relevance and frequency, and expectations that change is possible.

Little is known about M&M review in Canadian family practice. An unpublished 1998 study showed that only seven family medicine teaching sites responding to a single mailed survey reported doing M&M audits. During the last 4 years, we have conducted M&M audits in ambulatory family practice. This paper aims to contribute to the quality improvement literature by sharing, particularly with those who have little experience of audits, the challenges of M&M review in family practice. We describe the process of one M&M audit, practice changes resulting from it, and the strengths and limitations of the methods used. How this type of audit applies to practices of different sizes, in various locations, and with diverse remuneration systems will also be discussed.

Program

Creating a Morbidity and Mortality Review Committee.

St Mary’s Hospital in Montreal, Que, is a secondary care, McGill University–affiliated institution. Its Family Medicine Centre has about 18 000 active patients who make 27 000 visits per year. It is staffed by seven full-time and 22 part-time family doctors, five nurse practitioners, 36 family medicine residents, and 25 rotating medical and nursing students.

Our past attempts at M&M review met with reluctance from physicians because of their concerns about time commitment, remuneration, and what would be done with audit results. Other studies have reported fear of decreased clinical ownership or litigation, hierarchical and territorial concerns, uncertainty about sampling techniques, and questions about whether audits actually test what they are designed to test.^7,13 We moved ahead in response to an accreditation recommendation.

Six full- or part-time physicians served on the M&M review committee. They chose a chairperson from among themselves; the department chief acted as an advisor. The committee is remunerated through Quebec’s hourly payment system for teachers in family medicine centres. The committee spent much of the first year defining their mandate. Being relatively inexperienced in M&M review, we acknowledged from the start that we needed to experiment with how to approach such activities. We understood that quality indicators could be developed in either an unsystematic or a systematic, evidence-based way.⁵ We decided that audit results would be shared only after the committee was comfortable with its methods.

Making decisions.

There were concerns about both interreviewer and intrareviewer reliability and how to decide between using explicit criteria (ie, clearly predetermined standards, including clinical practice guidelines) or implicit criteria (implied, understood, or reflective of each doctor’s professional standards). In a rigorous analysis of peer review using both explicit criteria and implicit judgment, Hulka et al¹⁶ concluded that, at least in the context of ambulatory care, either method resulted in comparable findings. The authors indicated, therefore, that cost factors, feasibility, and acceptability could influence choice of method, and that reviewers need not choose one criterion over another because both could be used in a complementary fashion.

The Royal College of General Practitioners’ practice activity analysis¹⁷ suggested that use of standards based on the judgment of people possibly detached from the front lines of medical care might fail to recognize the working reality of general practitioners. Finally, recently published observations suggested that less structured peer review is particularly useful for outpatient settings.⁶

The goals for each audit were generated by the full committee. We started an audit with predominantly implicit criteria that we hoped were sufficiently inclusive to flag enough possible, probable, and clearly identifiable problems to stimulate discussion, learning, and recommendations. We made specific audit forms for each review (Figure 1), and at the beginning of each new audit, all committee members discussed common charts in order to establish consensus on what was being sought.

Figure 1.

Content of audit form

Later, charts were assessed by single reviewers seated at a common table where there was opportunity for collegial input on cases where problems were suspected. Each assessment received an evaluator’s subjective confidence rating (high vs low). If a disagreement could not be resolved, relevant clinical practice guidelines were consulted. When consensus could not be achieved or confidence levels remained low, no conclusion was assigned to the case. No final conclusions were drawn about any problems until treating physicians had an opportunity to comment on whether something had been missed or misinterpreted. Reports on completed audits were presented to the centre’s medical director. Recommendations were presented with “blinded” findings to doctors, nurses, trainees, and clerical personnel during meetings at the Family Medicine Centre.

Identifying preventable hospital admissions.

We thought the topic “preventable hospital admission,” cited in the literature as a care outcome indicator worth studying,¹⁸ was a good choice for an audit. Inclusion criteria were non-elective admissions of medical, surgical, psychiatric, or gynecologic cases. Exclusion criteria were cases receiving elective procedures and palliative, infertility, and intrapartum care (the last audited by the department of obstetrics).

Because of the common difficulty in being notified about hospitalized patients,¹⁹ we limited our review to admissions within our hospital where a computerized cross-referencing of our ambulatory patients with admitted patients gave us a complete database with which to work. Since discharge summaries rarely provide sufficient information for audits, we needed access to hospital charts, and those in our own institution were readily available. While this approach might have limited the generalizability of our audits, it did not invalidate what we could learn about our hospital sample.

Admissions for 5 randomly chosen months within the preceding 2 years were identified for paired review of office and hospital charts. Each such audit took an average of 15 minutes per physician; 30 minutes were needed when additional peer input was necessary. Of the 178 admissions considered, 75 met the inclusion criteria. Of these 75 admissions, 19 were found to be possibly preventable (Table 1). After subsequent review by the full committee, eight were upgraded to likely preventable. Nevertheless, because we thought there might be variability in ratings, we retained all 19 cases as the basis for recommendations for improvement.

Table 1.

Results of audit of possibly preventable admissions

Program evaluation

Committee members brought different expectations, interests, and expertise to the table. They needed time to have confidence in the process and to consider what our findings might imply. The literature suggests factors that facilitate audits include strong teamwork with shared dialogue, mutual confidence, collegiality (no hostility or threats), shared or rotating leadership, dedicated staff, protected time, structured programs, openness to advisors (if available), presence of a program manager, computerization (at least an age-sex registry), organized notes (including clinical and medication summaries), groups of three or more people, and practices that are involved in teaching.^{4,7,10,17,20,21} Preparing this paper helped us to recognize that these factors were present in our group.

Reported barriers to audits include lack of planning, resources, and expertise for project design and analysis; high costs for support staff, computer software, and data-collection sheets; and limitations on software functions.^7,22,23 Our initial review suggested that our limitations would be the relative unsophistication of our audits and the necessity of relying on written charts rather than computerized ones.

A strength of our M&M review committee was the heterogeneity of interests among the physicians who sat on it. Interests included emergency medicine, hospitalist practice, internal medicine, medical ethics, palliative care, and perinatology. While some of these areas were not included in our audits, knowledge of one area could often be generalized to another.

Length of time on committee.

Our experience suggests members should sit on the committee for at least 2 years. While some committee members seemed to be fatigued with the work in the third year, only one person asked to be replaced after more than 4 years on the committee. We speculate that by 4 years, most committee members were feeling a growing comfort with and acceptance of the task.

Payment for committee work.

Remuneration is traditionally a problem in getting physicians committed to quality improvement activities. While our payment system is not generous, it has been sufficient to keep most committee members involved. Other funding avenues could be explored through provincial governments or medical associations.

Frequency of audits.

British GPs have reported initiating an average of 3.5 (range one to seven) audits and completing an average of 0.9 (range none to three) audits during a 2-year period.²⁴ When 189 practices in Britain were assigned audit advisory groups, 169 initiated audits: 26% of these completed at least one audit, but 24% did not start any.¹⁰ Another report of GPs working with audit advisory groups showed a median of three audits conducted per practice over a 3-year period.¹¹ Our group has completed four different audits in 3.5 years, a somewhat better rate than that reported in Britain even though our work is being done without the benefit of an audit advisory group.

Benefits of our audits.

As a result of our findings on the audit of preventable hospital admissions, we have done the following.

• We now have a protocol for missed patient appointments that requires physicians to review the charts of patients who miss appointments and to give written instructions for follow up.

• We have added computers to our centre to minimize delays in obtaining laboratory results and to facilitate access to on-line clinical practice guidelines and other databases.

• We have developed new procedures for processing hard-copy laboratory results and getting them on file more quickly.

As 2 years have passed since we performed this audit, we are now repeating it to see whether the changes we made have addressed previously identified concerns. It might be difficult, however, to draw definitive conclusions because we do not have a stable cohort of staff at our Family Medicine Centre. Since it is a training site, we have an annual turnover of resident physicians.

Conclusion

Our M&M audit in family practice provided sufficient information to encourage practice improvement that would benefit both patients and physicians. While the science behind doing audits is becoming increasingly sophisticated, there is room for flexibility in how audits are approached when they are done in ambulatory care. Such flexibility might be necessary to encourage those inexperienced in performing audits to undertake them. While our reviews were done in the setting of a large teaching practice, we believe our approach could be adapted by physicians who practise solo or in small groups if they form larger working groups with other physicians in similar practices solely for the purpose of collegial practice audits.

Biography

Dr Yaffe is Chief of the Department of Family Medicine at St Mary’s Hospital in Montreal, Que; the other authors are members of the Family Medicine Centre at St Mary’s Hospital. Dr Yaffe is an Associate Professor, Dr Boillat is an Associate Professor and Postgraduate Program Director, and Drs Gupta, Still, Russillo, Schiff, and Sproule are Assistant Professors, all in the Department of Family Medicine at McGill University in Montreal.