Improving measurement of primary care system performance

Policy makers, primary care providers, and patients need high-quality information about the performance of their primary care system to enable informed decision making and efficient allocation of resources. When different jurisdictions measure the performance of their primary care sectors in similar ways, it is possible to determine which factors in primary care service delivery are associated with better outcomes. Better understanding of what works and what does not work can help to improve primary care systems, inform policy, and direct reform efforts. This paper focuses on how measuring, comparing, and reporting on the performance of the primary care sector can be improved, including what we know, the limitations of the data, and how we can go about getting the data we need. The intent of this work is to ultimately contribute to improved health status and longevity at the population level.

Conceptual framework

A conceptual model can help to ensure all the defining features of primary care are considered when developing indicators to measure and report on the performance of the primary care sector. In 1966, Donabedian published his now-classic work on assessing the quality of health care.¹ Guided by his framework, there have been various efforts over the past few decades to deconstruct the components of primary care.^2–6 Most of these models include components of structure, process, and outcome.¹ The Institute of Medicine in the United States developed a framework that has been used to plan reform in primary care and to create data collection tools to measure the quality of the delivery of primary care.^2,4,5,7–9 Starfield’s framework¹⁰ included structure, process, and outcome across categories of capacity, performance, and health status. Campbell et al² identified the need to distinguish quality at the individual level from quality at the population level. Others^7,11 have highlighted the range of organizational contributions made by governments to support primary care. The Organisation for Economic Co-operation and Development¹² identified both micro and macro factors at play, as well as the importance of a framework that addresses primary care’s complexity.¹³ Lamarche et al¹⁴ found that factors such as the environment of the practice are also key.

A conceptual framework for comparing models of primary care service delivery developed at the University of Ottawa in Ontario builds on this body of work.⁴ The most important contribution of this new conceptual model is the structural domain that describes the context in which care is delivered, including the broad overarching health care system, the regional practice context, and the local organization of the practice. Understanding the practice context and the socioeconomic profile of the patients is critical when interpreting performance measures. The Comparison of Ontario Primary Care Models project used this framework to determine which organizational factors were associated with better outcomes.¹⁵

Performance data

A conceptual framework is only the first step. For each dimension, indicators must be chosen to “measure” the quality of care delivery.^16–18 For example, the percentage of patient visits to emergency departments that could have been looked after in community-based practices is one indicator of access to primary care services.

Data that are available, including surveys of providers conducted by the Commonwealth Fund, show that Canada consistently ranks among the worst in developed countries in terms of primary care performance.¹⁹ For example, Canada (43%) and the United States (29%) are among the bottom 3 countries in terms of the proportion of practices that have arrangements for patients to access after-hours care with a physician or a nurse.²⁰ This is substantially less than in countries like the Netherlands (97%), New Zealand (89%), and the United Kingdom (89%). Canada has the lowest proportion (37%), followed by the United States (46%), of physicians using electronic medical records (EMRs).¹⁹

The most important limitation of primary care system performance data is the overall lack of such data.^20,21 This limitation is important because public debate on how to improve our health system is driven by the information that is available. Detailed performance data on other areas of the health system, such as the acute hospital sector, are available. Not surprisingly, public debate on hospital-related issues such as wait times for surgery is common, while a parallel debate on challenges in primary care is not taking place to the same extent.²² The Canadian Institute for Health Information (CIHI) recently released a 120-page report on health during the past 10 years in Canada, which acknowledges that “the effectiveness of primary care remains largely a black box.”²¹ The lack of performance measurement data for primary care contributes to the poor performance of the primary care sector in Canada.¹⁶ Development work on primary care indicators at the CIHI has started,²³ including the production of a list of 105 indicators (data on about 30 of which are routinely collected).

Another limitation is that the performance information being reported is the least expensive to collect and not the most important. For example, telephone surveys are often used, which have low response rates and rely on patients’ memories long after they have visited their providers. To get a comprehensive picture of primary care performance we need to add to existing telephone surveys of patients and providers.

Using health administrative databases in performance monitoring is attractive because the data are already routinely collected (and using data that are routinely collected is cheaper than collecting new data), they are population-based, and different components of care can be linked. However, data from health administrative databases might not be easy to compare across jurisdictions (eg, in Canada, each province measures similar aspects of care differently), and considerable information gaps exist for crucial aspects of primary care delivery (eg, in Canada, there are no data on services provided by non-physicians or on care received by aboriginal Canadians living on reserves or Canadians serving in our military). Linking pay-for-performance incentives to a quality-outcomes framework, as has been done in the United Kingdom^24,25 or the performance-and-diligence indicators used in Alberta,²⁶ further enriches health administrative databases. While they will play an important role in measuring dimensions such as relational continuity and the delivery of preventive services, they will never contain all the information in patients’ charts.

Abstracting charts can provide the most robust information for some dimensions of care, particularly technical aspects of care. Remote abstraction of information from EMRs might someday lower costs by replacing the need to travel to practices. In the short term, however, it is impossible to get a representative sample of practices or providers in countries like Canada where only 37% of primary care practices use EMRs.¹⁹ It might never happen that a representative sample of physicians agrees to provide remote access to or copies of their EMRs for performance measurement purposes. At least for now, it is necessary to travel to each practice to access the detailed technical quality-of-care data contained in the patient charts. Even on-site chart abstraction has its limits. It is not possible to retrieve good information if it is not reliably recorded (eg, information on health promotion).²⁷ For this and other dimensions, such as cultural competency, trust, and patient-centredness, it is necessary to deal with patients directly, which can also be done while visiting the practice.

Although visiting practices is very expensive, one key advantage includes the ability to randomly sample and recruit practices in a given jurisdiction. A research assistant then travels to the practice to complete the data collection using a series of approaches that provide a comprehensive picture. Data collection could start in the waiting room by surveying patients as they arrive and again immediately after they have seen their providers. Because the patient completes the survey immediately after seeing the provider, he or she can accurately remember what happened. Patient surveys allow the activities of all providers (not just the physicians) to be captured and provide the opportunity to get consent to link the data to health administrative databases and chart audits. When patients are approached in this way, they are often willing to share very sensitive data about themselves and their contexts, such as socioeconomic status.²⁸ In addition, the practice manager might be asked to complete a general questionnaire, and providers can complete a very brief data collection card for each patient encounter using techniques perfected by the National Ambulatory Medical Care Survey²⁹ and Bettering the Evaluation and Care of Health⁹ data collection systems in the United States and Australia. Finally an audit of patient charts, whether paper or computerized, can be conducted on site. All components of the data collection are then linked together. Sampling consecutive attendees in the practice waiting room has its limitations as well, of course. Patients who are sicker are more likely to be surveyed, as they visit the office more frequently.

Implementing a performance measurement system

The ethical and privacy issues to do this work can all be addressed, and such work has been done, albeit on a small scale.¹⁵ Government or quasi-government agencies that are responsible for data collection and analysis, such as the CIHI in Canada, are well positioned to take this work on. A performance measurement system of this scale is expensive, but necessary. The educational bodies for physicians, such as the College of Family Physicians of Canada, should be key partners in the development and implementation of a credible, relevant, objective, and transparent measurement and reporting system. The participation of these bodies would provide considerable reassurance to family physicians that performance measurement is being properly developed and conducted, and would encourage them to participate.

Public reporting of performance has been shown to lead to modest improvements in care.^30,31 System-level reporting can motivate change and investment.^32,33 Additional performance measurement data, including data directly from practices, needs to be collected across provinces and countries in a consistent manner to enable comparisons and to help determine the attributes of the best-performing primary care systems. Comprehensive performance information will help to build better primary care systems and ultimately improve health status and longevity at the population level.