Can the Quality of Care in Family Practice Be Measured Using Administrative Data?

Abstract

Objective

To explore the feasibility of using administrative data to develop process indicators for measuring quality in primary care.

Data Sources/Study Setting

The Population Health Research Data Repository (Repository) housed at the Manitoba Centre for Health Policy which includes physician claims, hospital discharge abstracts, pharmaceutical use (Drug Program Information Network (DPIN)), and the Manitoba Immunization Monitoring Program (MIMS) for all residents of Manitoba, Canada who used the health care system during the 2001/02 fiscal year. Family physicians were identified from the Physician Resource Database. Indicators were developed based on a literature review and focus group validation.

Data Collection/Extraction Methods

Data files were extracted from administrative data available in the Repository. We extracted data based on the ICD-9-CM codes and ATC-class drugs prescribed and then linked them to the Physician Resource Database. Physician practices were defined by allocating patients to their most responsible physician. Every family physician in Manitoba that met the inclusion criteria (having either 5 or 10 eligible patients depending on the indicator) was ‘scored’ on each indicator. Physicians were then grouped according to the proportion of the patients allocated to their practice who received the recommended care for the specific indicator.

Principal Findings

Using administrative health data we were able to develop and measure eight indicators of quality of care covering both preventive care services and chronic disease management. The number of eligible physicians and patients varied for each indicator as did the percent of patients with recommended care, per physician. For example, the childhood immunization indicator included 544 physicians who, on average, provided immunization for 65 percent of their patients.

Conclusions

Quality of care provided by family physicians can be measured using administrative data. Despite the limitations addressed in this paper, this work establishes a practical methodology to measure quality of care provided by family physicians that can be used for quality improvement initiatives.

Primary care is the foundation of the Canadian health care system. A strong Primary Health Care (PHC) system has been shown to result in a healthier population (Shi 1997; Macinko, Starfield, and Shi 2003) and may also contribute more to the health of the population than do specialized services (Starfield and Shi 2002; Baicker and Chandra 2004). Indeed, most health problems are initially treated by primary care physicians (i.e., family physicians or general practitioners; Green et al. 2001). An effective PHC system is necessary to address the challenges of an aging population, an increase in chronic disease, complex comorbidity, and/or functional disability in the population (Future of Family Medicine Project Leadership Committee 2004). Recognition of this need has led to efforts to improve the delivery of primary care in several countries to improve access and quality.

The Institute of Medicine has defined quality of care as “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge” (Lohr 1990, p. 128–129). Reviews of quality of care in general practice in the United States, the United Kingdom, Australia, and New Zealand have found established standards of practice to be rarely met (Seddon et al. 2001; McGlynn et al. 2003). In Canada, we know comparatively little about the quality of primary care. In an effort to improve this, the Institute of Health Services and Policy Research of the Canadian Institute of Health Research (CIHR) recently identified “Managing for quality and safety” as a priority area for research funding. The Institute expressed a strong interest in “research designed to identify management strategies to improve quality at an affordable cost and to support more extensive use of performance indicators …” (Canadian Institutes of Health Research 2004).

Fundamental to this process is the development of tools and methods to measure quality and performance (Donabedian 1980). For example, there has been recent interest in using indicators to measure quality of primary care (Seddon et al. 2001; McGlynn et al. 2003). Quality can be measured in terms of structures (characteristics such as personnel, equipment, finances), processes (the actual care given encompassing clinical and interpersonal effectiveness (Campbell et al. 1998), and outcomes (the consequences of care such as health status and user satisfaction; Donabedian 1980). While these three dimensions are somewhat interdependent, good quality in one area does not imply good quality in another (Gandhi et al. 2002).

In primary care, health outcomes are not appropriate measures of quality because they depend on factors unrelated to the health care system, such as socioeconomic status (Frohlich and Mustard 1996; Sheldon 1998; Martens et al. 2002), as well as upon the quality of care provided at all levels of care—primary, secondary, and tertiary. Instead, process measures are generally accepted as the most useful indicators of quality (Brook, McGlynn, and Cleary 1996); thus, the study described in this article sought to develop process measures, focusing specifically on the clinical care provided by physicians. These measures can be examined using different approaches. Surveys allow for the collection of data about both the process of care and satisfaction with care but are costly and subject to recall and nonresponse biases (Vogt et al. 2004). Medical record audits provide a more comprehensive view of process measures but are expensive, and can be limited by poor quality documentation in patient records (Marshall et al. 2003; Vogt et al. 2004). Using chart information in combination with data available from administrative databases also poses significant confidentiality and cost challenges in an environment where electronic charts are uncommon. Direct observation has been used as a research tool but is very expensive and potentially intrusive (Stange et al. 1998). Thus, these limitations make primary data collection unrealistic as a source of longitudinal data to monitor quality of care (Brook, McGlynn, and Shekelle 2000).

Administrative health data provide an alternative cost-effective source of information for health services research in general, and the development and/or measurement of primary care process indicators, in particular. Although these data result from the daily work of running a health care system rather than being collected for research purposes, they are readily available for each full year at an individual level, thereby allowing researchers to examine important patterns of “health care service use, expenditures, selected clinical outcomes, and quality of care” across various health care settings over time (Iezzoni, Shwartz, and Ash 2005, p. 141). In the United States, such data are available from public and private insurers but only for select groups of people, according to the scope of the particular insurer. They also vary in terms of content and format. Canada's health care system, however, provides all residents with first dollar coverage for all medically necessary physician and hospital services.

Each province is responsible for the administration and delivery of health care to its residents within the parameters set out in the Federal Canada Health Act. Thus, provincial administrative data stem from a single source and cover the entire population. In Manitoba, almost every contact the population has with the health care system is recorded in the Province's database for administrative and billing purposes (Black, Roos, and Roos 2005). These data are collected from physician billing claims, from retail pharmacies for all prescriptions dispensed in the community, from hospital abstracts submitted for all day surgery and inpatient stays, from the Home Care program regarding receipt of services, and for all personal care (nursing) home admissions. Physician claims are submitted by both fee-for-service physicians (who submit claims to the Manitoba government for remuneration), and by physicians paid completely or in part via alternate payment mechanisms (e.g., salaried, contract); claims submitted by the latter group (called shadow billings) are used only for administrative purposes. Fee-for-service physicians have been shown to reliably submit claims for the services they provide, but this has not been established for physicians on alternate payment plans. Most family physicians (80 percent) in Manitoba bill fee-for-service (Katz et al. 2004; Watson et al. 2004). Optometrists are also entitled to bill for “medically necessary” services such as eye exams for diabetic patients. All other services are billed directly to the patient; these other services do not appear in the administrative data.

Manitoba Health assigns a unique numeric identifier to every person registered for public health insurance in Manitoba, that allows for them to be tracked across sectors of the health care system and longitudinally. Care provided to Manitobans by out-of-province physicians and care provided by Manitoba physicians to out-of-province residents are also tracked in a separate file. However, as these “reciprocal” claims represent less than 1 percent of all claims and our focus was on the regular care provided by Manitoba physicians to in-province residents, we chose to exclude these claims. Hence, administrative health data are a rich source of information collected consistently over a long period of time that readily lend themselves to population-based research. Several provinces across Canada—notably, Manitoba, British Columbia, Ontario, and Quebec—are already engaged in such research (Tamblyn et al. 1995; Roos, Menec, and Currie 2004).

The work described in this article was part of a larger study. The goal of the study was to explore the feasibility of using administrative data to develop process indicators for measuring quality in primary care in Manitoba. Our two key objectives were to: (a) develop indicators acceptable to practicing family physicians; and (b) describe the quality of care provided by Manitoba physicians using the selected indicators. In this article we focus on the first objective.

METHODS

Our research design followed three steps. First, we developed process indicators based on a review of the literature, the feasibility of using administrative data, and the input of physician focus groups. Second, the indicators were defined for measurement using the available administrative data. Next, physician practices were defined by allocating patients to their most responsible physician. Finally, every family physician in Manitoba that met the inclusion criteria was “scored” on each indicator. Physicians were then grouped according to the proportion of the patients allocated to their practice who were eligible for the specific indicator. Each of these steps will be detailed below. The process we followed to define physician practices and measure the indicators is illustrated in Figure 1.

Figure 1.

Method for Defining Physician Practices and Measuring Indicators.

Study Setting and Data Sources

The analyses used Manitoba administrative data available in the Population Health Research Data Repository (Repository), housed at the Manitoba Centre for Health Policy (MCHP). Previous studies have established the high quality of the Repository data for ambulatory care compared with other data sources (Hux et al. 2002; Roos et al. 2005). The Repository contains the anonymized records for all Manitobans' contacts with the health care system; there are approximately 1.1 million people in Manitoba, of whom 60 percent live in Winnipeg. Specific data used were from physician claims, hospital discharge abstracts, pharmaceutical use (Drug Program Information Network [DPIN]), and the Manitoba Immunization Monitoring System (MIMS). Before data transfer, Manitoba Health processes the records to encrypt all personal identifiers and remove all names and addresses. Physicians identified as general practitioners (GPs) in the Physician Resource Database were included. Before transfer of these data from Manitoba Health, physicians' personal identifiers are removed and billing numbers are encrypted to ensure their confidentiality. Physician claims include tariff codes used by fee-for-service physicians for remuneration, and one “most responsible” ICD-9 diagnosis code. Tariff codes are the Manitoba equivalent of Current Procedural Terminology (CPT) codes developed by the American Medical Association. Hospital discharge abstracts include up to 16 diagnosis codes. Data from fiscal year 2001/02 were used; the childhood immunization and postmyocardial infarction (β-blocker prescribing) indicators also required data from fiscal years 1999/2000 and 1999/2000–2000/01, respectively.

Indicator Development

A literature review identified indicators of quality in family practice that are associated with positive health outcomes. Several of the indicators used in our study are also part of a set of health measures used in other indicator development initiatives (e.g., the Health Plan Employer Data and Information Set [HEDIS] in the United States and the National Health Service [NHS] in the U.K.) (National Committee for Quality Assurance 2002; Marshall et al. 2003). This list was then limited to those potentially measurable with the available administrative data.

To ensure that our indicators were acceptable to practicing physicians the resulting 16 potential indicators were presented to focus groups of family physicians at three clinics (two in Winnipeg, one in rural Manitoba). Clinics were purposefully selected to include a range of characteristics such as region, the socioeconomic status of the area, physician gender, and practice structure. All three clinics had more than six full-time family physicians, an accessible physician contact to arrange the groups, community-based nonacademic physicians, and physicians with and without hospital privileges. All physicians at the clinics were invited to participate; six to 10 physicians attended at each site. Each participant received the list of potential indicators along with a brief explanation of each one and its intended use. After independent review, they engaged in a group discussion (facilitated by the researchers) about each indicator's validity, relevance to their practice, and any concerns. Dialogue continued until consensus was reached. The results of each session were presented to subsequent groups during their discussions.

Definitions for some indicators were modified as a result of the focus group feedback. For example, the observation period for some indicators was lengthened to allow detection of target behavior that occurred later but still within an acceptable time frame (Vogt et al. 2004). Two indicators (spirometry for asthma care and PSA testing) were suggested by focus group participants but omitted after exploring the feasibility of capturing the necessary data. Other initially identified indicators (antibiotic prescribing rates, consultation rates, and thyroid functioning screening/testing) were subsequently excluded as no benchmarks were available for comparison. The final list included 13 indicators. Data for five of these indicators (cholesterol screening, blood sugar screening, anticoagulant medication management, cholesterol testing for diabetic patients, and postmyocardial infarction cholesterol testing) are only available for Winnipeg, and were therefore excluded from this article, which focuses on indicators applicable to the entire province. Eight indicators are discussed here (see Table 1).

Table 1.

Indicators of Quality Primary Care

Indicator	Definition
Preventive services
1. Childhood immunization	Eligibility: Patients born in 1999
	Recommended care:% who received their primary course of immunization (i.e., DPT-HiB polio × 4, and MMR) by age 24 months
2. Influenza vaccination	Eligibility: Patients aged 65 years or older in 2000/01
	Recommended care:% who received at least one influenza vaccine between fiscal years 2000 and 2001
3. Cervical cancer screening	Eligibility: Female patients aged 18–60 years in 2001 who had not undergone a hysterectomy
	Recommended care:% who had at least one Papanicolaou test in the last years
Chronic disease management
4. Antidepressant medication management	Eligibility: Patients with a new prescription for an antidepressant within 2 weeks of a depression diagnosis
	Recommended care:% who had three subsequent ambulatory visits within months of the prescription being filled
5. Asthma care	Eligibility: Patients with an asthma diagnosis (defined as those who filled at least two prescriptions of a β2-agonist in the study year)
	Recommended care:% who filled a prescription for medications recommended for long term control of asthma (i.e., inhaled corticosteroids or leukotriene modifiers)
6. Diabetes care: eye examination	Eligibility: Diabetic patients (defined as those who filled at least two prescriptions for at least one drug used to treat diabetes)
	Recommended care:% who saw either an optometrist or ophthalmologist in the same fiscal year as the prescription
7. Postmyocardial infarction care: β-blocker prescribing	Eligibility: Patients discharged alive from hospital between fiscal years 1999 and 2001 with a discharge diagnosis of myocardial infarction (excluding those with a prior diagnosis of asthma, COPD or peripheral vascular disease)
	Recommended care:% who filled at least one prescription for a β-blocker within four months of the hospital discharge
8. Potentially inappropriate prescribing of benzodiazepines for older adults*	Eligibility: Patients aged 75 years or older in 2001
	Recommended care:% who filled prescription(s) for either two or more benzodiazepines or for greater than a 30-day supply of medication

Note: To be considered eligible, patients had to have at least one physician visit in 2001/02.

Note: The ICD-9-CM and Anatomical Therapeutic Chemical (ATC) codes used to define the conditions of interest are presented in the original report, which is available online at http://www.umanitoba.ca/centres/mchp/reports.htm.

^*For this indicator, a lower value is more desirable.

DPT-HiB, diphtheria, pertussis, tetanus, Haemophilusinfluenza B; MMR, measles, mumps, rubella.

Practice Populations

In Manitoba, where access to physicians is not formally restricted, patients may consult with any primary care physician. As a result, patients tend to visit different physicians over time. In the year before the study, Winnipeg residents visited an average of 1.9 family physicians, with those making more than 10 visits attending an average of 3.6 physicians (Watson et al. 2003). Thus, in order to reflect the care patients receive as a function of physician behavior, we needed to first define a practice population for each physician (see Figure 1).

The physician visits data file provides a record of each patient's visit to a physician every year. Using these data, we assigned patients visiting at least one physician in 2001/02 to the physician most responsible for their primary care. To select the most appropriate approach to patient allocation, we compared four methods, using physicians with at least 1,000 visits: (1) allocating to the physician with the greatest number of visits (ties were broken by arbitrarily allocating to the physician with the lowest billing number); (2) allocating to the physician with the greatest monetary value of visits (ties were broken by allocating to the physician with the lowest billing number); (3) allocating to the physician with the greatest monetary value of visits (ties were broken by allocating to the physician with the highest total costs which include visits and referrals for other services such as laboratory and imaging services, and consultations with specialist physicians); and (4) allocating to the physician with the greatest total costs (as defined in the previous approach) (see Appendix A of Katz et al. 2004). For each approach, we correlated the allocation results with the observed visits and observed patients. The third approach was chosen based on the correlations of 0.96 for visits and 0.79 for patients. This approach assumes that the assigned physician bears overall responsibility for that patient's primary care.

Once a patient was allocated to a physician, all relevant services the patient received were credited to that physician regardless of who provided those services. Thus, all relevant visits, immunizations, drug prescriptions, or laboratory tests ordered by any physician for that patient, or services provided by another health professional such as a public health nurse (e.g., immunization), were credited to the assigned primary care physician. For example, a patient who received 60 percent of care (based on expenditure) from physician A, and the remaining 40 percent from physician B was assigned exclusively to the practice of physician A. Physician A “benefits” from any appropriate services provided by Physician B, but is also “penalized” for any undesirable services provided by Physician B. This approach is consistent with the ultimate goal, which is to ensure the patient received the recommended service.

Measuring the Indicators

For every indicator, we calculated the ratio of all patients within each practice who met the specified eligibility criteria (denominator) to those who received the recommended care (numerator). This ratio became the physician's score. The cohort of physicians for each indicator included only those with a sufficient number of eligible patients. For most indicators, the minimum requirement was 10 eligible patients. For three indicators—childhood immunization, antidepressant medication management, and postmyocardial infarction β-blocker prescribing—the minimum was five because of relatively small overall numbers per physician for the province. The indicator definitions, including patient eligibility criteria and recommended care are described in Table 1. The eight indicators of quality were divided into two categories—preventive care services and chronic disease management. The criteria and the ICD-9 and Anatomical Therapeutic Chemical (ATC) codes used to define the conditions of interest are available on Manitoba Centre for Health Policy's website at http://www.umanitoba.ca/centres/mchp/reports.htm.

RESULTS

Table 2 presents, for each indicator, the number of physicians and patients included at each stage and the analyses results. A total of 952 out of a possible 997 Manitoba physicians met the minimum requirement for at least one indicator; the number of physicians varied across the indicators. A high proportion of physicians who had at least one eligible patient also met the minimum requirement (five or 10 patients) to be included (ranging from 55 percent for postmyocardial infarction to 94 percent for cervical cancer screening). Hence, for childhood immunization, of the 796 physicians with at least one eligible patient, 544 (68 percent) met the requirement of having at least five eligible patients. These physicians had a total of 9,532 eligible patients. Of these, 6,200 (65 percent) received their primary course of immunization by 2 years of age.

Table 2.

Percent Patients with Recommended Care per Physician

	#Physicians	Eligible Patients*	Mean % Patients with Recommended Care, per Weighted Physician (95% CI)†
Childhood immunization
Eligible‡	796	10,100
Cohort§	544 (68%)	9,532 (94%)	65 (33–97)
Influenza vaccination
Eligible	963	136,398
Cohort	843 (88%)	135,954 (99%)	57 (33–81)
Cervical screening
Eligible	997	270,441
Cohort	933 (94%)	270,213 (99%)	69 (45–92)
Antidepressant follow-up
Eligible	864	11,363
Cohort	618 (72%)	10,774 (95%)	48 (18–78)
Asthma care
Eligible	930	29,119
Cohort	728 (78%)	28,255 (97%)	61 (38–83)
Diabetes: eye exam
Eligible	923	33,326
Cohort	703 (76%)	32,400 (97%)	38 (16–59)
Post-Ml: β-blocker
Eligible	783	5,568
Cohort	431 (55%)	4,763 (86%)	60 (27–92)
Benzodiazepine prescribing
Eligible	933	74,161
Cohort	762 (82%)	73,456 (99%)	16 (3–28)

^*These patients met the indicator's eligibility criteria (denominator).

^†The mean proportion of patients per physician (not a population mean) is weighted by the number of eligible patients per physician. This places greater emphasis on practices with larger numbers of eligible patients and increase the validity of the confidence intervals.

^‡These physicians had at least one eligible patient.

^§These physicians had the required number of eligible patients and were scored on the indicator.

Figure 2 presents the distribution of physicians according to the proportion of their patients that received the recommended care for each of the quality indicators. For example, 49 percent of the physicians included in the childhood immunization indicator (n =265) fully immunized between 50 percent and 74 percent of their eligible patients.

Figure 2.

Distributions of Physicians by Percent Eligible Patients Receiving the Recommended Care.

DISCUSSION

This study demonstrates the feasibility of using administrative data to measure important components of clinical effectiveness in primary care. A key strength of this approach is the completeness of the data. The Repository contains anonymized information for every individual registered with the provincial health care program, allowing us to accurately reflect current Manitoba practice. The ability to link across files and over time using a unique personal identifier permits tracking registrants' encounters with various sectors of the health care system longitudinally, while ensuring the privacy and confidentiality of all information. Physician anonymity is also protected throughout the process. Compared with other potential data sources such as direct observation of patient–physician interaction, patient surveys, and clinical chart audits, administrative data have distinct advantages for measuring and monitoring quality of primary care over time. For example, while chart audits are limited to information from physicians whose charts are being reviewed, we were able to include all relevant care regardless of who provided the care. Provincial preventive health programs aimed at increasing patient coverage actively monitor service provision by all providers, including public health nurses (MIMS and cervical cancer screening), and sending reminders for delayed services (MIMS) enhance the completeness of these data. The characteristics and research benefits of a rich, comprehensive data system are elaborated in Black et al. (2005).

Another strength of our methodology is the ability to report results from the physician perspective by defining their practice populations, in contrast to health services research which traditionally reports population-based results. As shown in Figure 2, this approach allowed us to compare the variability within each indicator. Our data revealed room for improvement across the spectrum of services and conditions included. Even though the definitions are based on minimum requirements, the number of physicians with most of their patients meeting the target was still low for most indicators. In particular, for childhood immunization and post-MI β-blocker use, the quality of care provided by most Manitoba physicians was either below that published from other jurisdictions or did not meet national targets (Public Health Agency of Canada 1997; Tran et al. 2003).

Practicing physicians have little faith in the accuracy of the ICD-9 diagnoses entered for billing purposes. Most patients with chronic disease have more than one relevant diagnosis and the diagnosis recorded on the billing for any particular visit may not be the primary reason for the visit. Furthermore, many physicians informally report paying little attention to the accuracy of the diagnosis recorded, relying instead on codes committed to memory from frequent use. By combining prescription drug data with the ICD-9 diagnoses in our indicator definitions, we were able to confirm the validity of any diagnosis being attached to a patient. As our indicators are generally not visit-specific, the reason for any one visit is no longer of concern.

Some significant gaps in the data should be addressed if administrative data are to be used in quality improvement. Physician data derive from billing claims, which are reliable for services provided by fee-for-service physicians (Roos et al. 1993; Muhajarine et al. 1997). Reliability of the shadow billings by physicians under alternate payment plans, however, is not well-established. If there is underbilling by these physicians, our analyses may show physicians to provide poorer quality care than is the case. This issue is particularly relevant for rural areas as most of the non–fee-for-service physicians practice outside of the larger urban areas in Manitoba.

Some indicators involving laboratory tests performed outside Winnipeg or in Winnipeg hospitals, such as records for glycosylated hemoglobin and proteinuria tests (fundamental for the monitoring of diabetes), are not part of the provincial database. Central reporting for such services regardless of where they are conducted would broaden the scope of the quality indicators that could be examined. Other equally important aspects of quality, such as interpersonal effectiveness are not amenable to measurement using administrative data.

Quality indicators highlight areas with good quality care and those with potential problems. Our approach also creates the opportunity for providing physicians with individual, practitioner-specific feedback compared with the performance of their peers. Marshall et al. (2003) provides an extensive discussion of both the benefits and limitations of indicators, in general, while Sheldon (1998) and Epstein, Lee, and Hamel (2004) have addressed the advantages and disadvantages of providing individual feedback. While indicators can facilitate comparisons (among practices, over time, and against standards), promote accountability, and identify unacceptable levels of performance, they may focus solely on measurable components of care to the detriment of others. The benefits and limitations must be considered when developing and applying indicators (Hoey 2004). Used individually, indicators do not reflect the overall quality of care provided by a physician. A composite measure, however, could summarize quality across many indicators simultaneously but is beyond the scope of this article.

The methodologies described in this article provide the opportunity to measure the care provided by family physicians using administrative data in a nonrostering environment. We established these methods based on 1 year of data, but they can be easily applied to multiple years in other jurisdictions where comparable data are available over time, and may be used as part of a quality improvement initiative. Additional analyses should also explore the effects of patient and service provision (both individual physician and practice resource) characteristics upon the quality of care physicians provide and on the resulting health outcomes (Fiscella et al. 2000; Clancy 2005).