Identifying performance indicators for family practice

Abstract

OBJECTIVE

To identify performance indicators for family practice that focus on organizational structures and clinical processes of care, to review evidence linking indicators to patient outcomes, to have providers select indicators they consider important for performance assessment, and to obtain provider views on challenges to developing a performance assessment system.

DESIGN

Review of published and unpublished literature and contact with international experts resulted in a list of 131 structure and process indicators and associated evidence. This information was used in a two-round modified Delphi consensus process, which was followed by interviews with each of the 12 consensus panel members.

SETTING

Ontario family practices.

PARTICIPANTS

Eleven family physicians and one nurse practitioner from Ontario.

MAIN OUTCOME MEASURES

Survey package with 131 indicators and associated evidence was mailed to panel members who rated each of the indicators on a Likert scale from 1 (not at all important for performance assessment) to 9 (essential for performance assessment). Interviews were conducted with panel members to discuss indicator feasibility and data sources. Consensus score and median importance score for each indicator were main outcome measures; interviews identified barriers to performance assessment.

RESULTS

Fifty-one indicators achieved high consensus, 19 moderate consensus, and 38 low consensus. Clinical indicators that reached a high level of consensus were generally supported by grade A or B recommendations and level I to III evidence. Clinical indicators that achieved moderate consensus often had fair support in the literature. Low consensus was mainly associated with fair or equivocal evidence. During follow-up interviews, consensus panel members voiced frustration with inconsistencies in the evidence and practice guidelines upon which indicators are often based, and with poor transfer of patient information between health care providers. Lack of detail in patient care documentation and inconsistent documentation were mentioned frequently as threats to data quality.

CONCLUSION

Despite challenges to performance measurement noted by the panel, study results support the continued development, refinement, and testing of primary care performance indicators.

EDITOR’S KEY POINTS.

• Identification of performance indicators is important for evaluating quality of health care.

• In this Ontario study, 11 family physicians and one nurse practitioner rated performance indicators measuring disease prevention and health promotion, care for acute and chronic diseases, organizational features and practice management, and patient-provider relationships.

• Fifty-one indicators achieved high consensus, 19 achieved moderate consensus, and 38 achieved low consensus. Levels of consensus generally followed levels of evidence.

• Identification of indicators is a dynamic process that is determined in large part by information from ongoing scientific research.

• L’identification d’indicateurs est un processus dynamique qui doit s’ajuster en fonction de l’évolution des évidences scientifiques..

POINTS DE REPÈRE DU RÉDACTEUR.

• L’identification d’indicateurs de performance est importante pour l’évaluation de la qualité des soins.

• Cette étude ontarienne incluant 11 médecins de famille et une infirmière a vérifié le niveau de consensus de ces intervenants en regard d’indicateurs de performance mesurant la prévention la promotion de la santé, les soins pour des maladies aiguës ou chroniques, l’organisation et la gestion de la pratique, et les interactions entre les dispensateurs de soins et les patients.

• Le consensus a été élevé pour 51 indicateurs et modéré pour 19 autres. Les indicateurs cliniques sont habituellement soutenus par de meilleures évidences.

A study of Canada’s physician work force reported that reductions in physician supply since 1993 are the result of an increase in the time spent in postgraduate training, a decrease in intake of international medical graduates, an increase in retirements, and a change in physicians’ practice patterns.¹ The Romanow² and Kirby³ reports describe provincial initiatives under way in response to the reduction in physician supply and other pressures on primary care. Both reports endorse strengthening primary care through alternative funding models, increased health promotion and prevention activities, and greater coordination and integration of health care services.

To assess the effect of current and future renewal activities, rigorous and objective methods for measuring quality of care and other aspects of performance are needed. Such methods should describe current evidence supporting performance measures, include practitioners’ views on the relevance and feasibility of proposed approaches, and be based on data that are accurate and consistent across practices. Ideally, evidence supporting performance measures will come from well designed studies that verify relationships between primary care structural characteristics (eg, organizational features and practice management), clinical processes (eg, prevention and promotion, acute and chronic care, patient-provider relationships), and patient outcomes (eg, patient health status and satisfaction).

To date, systematic evaluation of the links between organizational structures, clinical processes, and patient outcomes has been limited. Thus, assumptions about relationships are often based on descriptive studies, expert opinion, and professional consensus. Data quality, a critical component of performance measurement, is seldom explicitly addressed. Data quality should be assessed, data limitations described, and recommendations made for improving quality.

This performance measurement study had four objectives:

• to identify performance indicators for family practice that focus on organizational structures and clinical processes of care associated with patient outcomes;

• to review the evidence that supports each indicator;

• to obtain providers’ views on which indicators are important in assessment of clinical and management performance; and

• to obtain respondents’ views on challenges to using indicators in family practice, including the issue of data quality.

METHOD

Ethics approval was obtained from the University of Toronto. In this study, primary care is defined as family and general practice in Ontario.

Performance indicators and supporting evidence

Key words and databases (Table 1) were used to search for articles on performance measurement in primary care published between 1990 and 2001. This search was supplemented by searches of professional association and federal and provincial government websites. Additional references were extracted from a continuously updated in-house bibliographic database containing citations on such topics as access, continuity, and comprehensiveness. National and international primary care experts also were consulted.

Table 1.

Key words and databases used in literature search

We created a manual that included indicators and associated evidence. When available, we provided classifications of recommendations from The Canadian Guide to Clinical Preventive Health Care⁴ or the levels of evidence proposed by D’Agostino and Kwan⁵ (Table 2). This information was not uniformly available, however, for all indicators.

Table 2.

Levels of scientific evidence for outcomes

*Levels 1 to 5 are derived from D’Agostina and Kwan.⁵

†Level 6 was added by the research team.

Selection of indicators

A Primary Care Panel (PCP) comprising 11 family physicians and one nurse practitioner, all in active practice, was established to evaluate the importance of the performance indicators. Names of possible panel participants were obtained from project team members, primary care opinion leaders in Ontario, and the Ontario Ministry of Health and Long-Term Care. Participants represented diverse geographic regions (urban, suburban, and rural), practice types and reimbursement methods (group and solo fee-for-service, globally funded community health centres, capitation-based health service organizations), and settings (community and academic).

A modified Delphi technique was used to establish consensus among PCP members on the importance of each indicator for performance assessment in family or general practice. This technique provides experts with the available evidence upon which to base decisions, combined with repeated rounds of a survey, and summary reports back to participants after each round.⁶ Individual responses are anonymous, thus avoiding influence from peerpressure.⁷ Our approach departed from the original Delphi technique in that our PCP did not convene face-to-face to discuss the indicators to be assessed. Our approach has been used in several health care studies.^6-8 A recent study, comparing convened panels to panels that participated only by mail, reported substantial agreement between the two panels with agreement on about 68% of the scenarios rated and serious disagreement on only 1% of the ratings.⁹

On December 15, 2000, PCP members received the Round 1 consensus package, which included a booklet containing instructions, a questionnaire with indicators to be rated, a description of current evidence for each indicator, and an addressed return envelope. Each indicator was rated on a Likert scale from 1 (not at all important for performance assessment) to 9 (essential for performance assessment). Panel members were invited to revise the wording of indicators, add and rate additional indicators, and provide other written comments.

Indicator ratings were entered into an Excel spreadsheet and independently checked by a member of the research team (J.B.). Median scores and frequency of responses in tertile Likert categories (1-3, 4-6, 7-9) were calculated.

Round 2 commenced on January 26, 2001. Panel members received a package similar to that sent in Round 1 as well as a report of first-round results that included, for each indicator, the individual member’s score, the median score, and the distribution of scores for the entire panel. Consensus for Round 2 was defined as follows:

• high consensus: a minimum of 10 responses in a single tertile and the remaining responses in an adjoining tertile;

• moderate consensus: a minimum of eight responses in a single tertile and at least two responses in an adjoining tertile;

• low consensus: indicator did not meet the criteria for high or moderate consensus.

Panel members were sent a report of Round 2 results. Individual semistructured interviews were held with PCP members between February 27 and March 27, 2001, to discuss the results of the consensus process, the feasibility of data collection, and possible data sources for indicators that reached high or moderate consensus. All interviews were conducted by members of the research teams: nine in person and three by telephone.

RESULTS

Consensus process

All 12 PCP members participated in Round 1 and Round 2 of the consensus process and the follow-up interviews. Tables 3 and 4 present high-consensus and moderate-consensus indicators and the median importance score for each indicator. Fifty-one indicators achieved high consensus, 19 achieved moderate consensus, and 38 had low consensus (not shown in Tables 3 and 4).

Table 3.

High-consensus indicators from Primary Care Panel consensus process

*1—not at all important; 9—essential for performance assessment.

Table 4.

Moderate-consensus indicators from Primary Care Panel consensus process

*1—not at all important; 9—essential for performance assessment.

Clinical indicators that received a high level of consensus tended to be supported by grade A or B recommendations and level I to III evidence. High consensus also was achieved for non-clinical indicators related to practice organization and management and patient-provider interaction. While these indicators have face validity, supporting evidence was either absent or weak, based on descriptive studies or expert opinion. Forty-two of the high consensus indicators had a median score of 8 or 9; the remainder had a median score of 7. High scores indicate that PCP members agreed that these indicators represent important aspects of performance in family practice. Only one acute indicator achieved high consensus: “Record that follow up has been done for abnormal urinalysis results.”

Clinical indicators that achieved moderate consensus tended to have fair support in the literature. All but one of the moderate consensus indicators had median scores of 7 or 8, suggesting that the items represent important aspects of performance. The one exception was the acute care indicator, “Successful treatment of UTI is routinely confirmed by a repeat urinalysis,” which received a median importance score of 2. This “negative” indicator was included as an indicator of inappropriate care or poor performance.¹⁰ Moderate consensus also was reached for non-clinical indicators for which supporting evidence was either absent or weak, based on descriptive studies or expert opinion.

The 38 low-consensus indicators were mainly associated with fair or equivocal evidence (eg, grade B or C recommendations from the Task Force on the Periodic Health Examination); 29 of these indicators had median importance scores lower than 7.

Follow-up interviews with consensus panel

During follow-up interviews, consensus panel members voiced various frustrations associated with performance indicators. These included finding inconsistencies in supporting evidence and practice guidelines and the failure of other health care providers to communicate information on patient care to family physicians for inclusion in patient records. Examples of the latter included information on prescriptions, immunizations and vaccinations, patient education and counseling, mammography, eye examinations, and investigations conducted in hospitals.

Lack of detail in recording processes of care, and inconsistencies among physicians in recording, were mentioned frequently by PCP members as threats to data quality. The former prevent an accurate portrayal of a physician’s care within a practice and the latter undermine comparisons between physicians and practices. Panel members identified indicator activities that might be carried out but not recorded. These included counseling on tobacco cessation or exercise, assessment of a patient’s emotional state, patient adherence to a drug regimen for hypertension, and counseling and education of patients with asthma. Panel members suggested that actions that identify problems are more likely to be recorded.

DISCUSSION

This study identified a range of indicators representing various aspects of family practice: prevention, promotion, acute care, chronic care, interaction with patients, and practice organization and management. Among these, clinical indicators are the most likely to have their association with patient outcomes investigated in rigorously designed studies. There are few studies and, thus, little evidence of the effect of practice management indicators on patient outcomes; several of these indicators, however, reflect standards related to workplace health and safety or access for those with physical disabilities.

Evidence of the link between patient-provider interaction and patient outcome is inconsistent.^11,12 Many of these indicators received high scores for importance, suggesting fertile ground for research.

Few indicators were selected for acute conditions. This might reflect a lack of acceptable guidelines, inconsistent guidelines, or poor dissemination of guidelines for these indicators.¹³ Further, it is unclear whether the negative indicator, “Successful treatment of UTI is routinely confirmed by a repeat urinalysis,” which achieved moderate consensus, was rated of low importance because it was considered a poor measure of performance or because it was considered a measure of poor performance. To avoid this problem in future, we will provide explicit instructions reminding consensus panel members that they are not being asked to evaluate the appropriateness of the practice described in negative indicators but rather to evaluate negative indicators in terms of their usefulness in identifying inappropriate care.

The frustration with guidelines mentioned by panel members is mirrored in the literature. For example, McAlister and colleagues¹⁴ noted that guidelines are inconsistent regarding the acceptable observation period before diagnosis of hypertension or blood pressure levels at which drug treatment should be initiated. Guidelines have produced inconsistent recommendations on the frequency of glycosylated hemoglobin tests for monitoring glucose control in diabetic patients,¹⁵ and frequency of fundoscopic eye examination for diabetic patients.¹⁶ In addition, guidelines and associated indicators do not always provide the detail and precision required for translation into the specific data elements that constitute quality measures.¹⁷ In such situations, clinical judgments must be made by those developing the measures.¹⁸ There is some controversy over the validity of the consensus process, including concerns that anonymity of responses might lead to poorly considered responses¹⁹ and the possibility of biased results depending upon selection of participants or the methods used to quantify data.²⁰ We included panel members from a variety of practice types and locations and presented participants with the median rather than the mean score and the range of panel scores to give a full picture of results.

It is impossible to identify a static set of indicators, as evidence changes constantly. For example, new evidence has been established for two low-consensus indicators (obtaining a fasting lipid profile²¹ and the use of ASA for prevention²²) for which evidence was inconclusive at the time of our consensus process. The importance scores for these indicators might be different if the study were conducted today. This example illustrates the necessity of having a dynamic performance measurement system that captures new evidence while allowing time for the evidence to be translated into practice.

CONCLUSION

The PCP reached high or moderate consensus on the importance of 70 primary care indicators related to prevention, promotion, acute care, chronic care, practice management, and interaction with patients. Despite the challenges described in the literature and by our expert panel, study results support the continued development, refinement, and testing of primary care performance indicators. To this end, a field test of indicators identified by the PCP has recently been completed in 10 Ontario family practices.

Biography

Dr Barnsley is an Associate Professor, Dr Berta is an Assistant Professor, Dr Cockerill is a Professor, Ms MacPhail is a Research Associate, and Dr Vayda is a Professor Emeritus in the Department of Health Policy, Management and Evaluation at the University of Toronto in Ontario.