Are quality improvement plans perceived to improve the quality of primary care in Ontario?

Abstract

Objective

To explore primary care administrators’ perceptions of provincially mandated quality improvement plans, and barriers to and facilitators of using quality improvement plans as tools for improving the quality of primary care.

Design

Qualitative descriptive study using semistructured interviews.

Setting

Ontario.

Participants

Eleven primary care administrators (ie, executive directors, director of clinical services, office administrators) at 7 family health teams and 4 community health centres.

Methods

All interviews were audiotaped and transcribed verbatim. Data were analyzed deductively to generate a framework based on a conceptual model of structural, organizational, individual, and innovation-related factors that influence the success of improvement initiatives and, inductively, to generate additional themes.

Main findings

Provincially mandated quality improvement plans seem to have raised awareness of and provided an overall focus on quality improvement, and have contributed to primary care organizations implementing initiatives to address quality gaps. Four factors that have contributed to the success of quality improvement plans relate to attributes of the quality improvement plans (adaptability and compatibility) and contextual factors (leadership and organizational culture). However, participants expressed that the use of quality improvement plans have not yet led to substantial improvements in the quality of primary care in Ontario, which may be owing to several challenges: poor data quality, lack of staff and physician engagement and buy-in, and lack of resources to support measurement and quality improvement.

Conclusion

Awareness of and focused attention on the need for high-quality patient care may have increased, but participants expressed that substantial improvements in quality care have yet to be achieved in Ontario. The lack of perceived improvements is likely the result of multifaceted and complex challenges primary care organizations face when trying to improve patient care. To effect positive change, organization- and health system–level efforts are needed to improve measurement capabilities, improve staff and physician engagement, and increase capacity for quality improvement among organizations.

In Ontario, the Ministry of Health and Long-Term Care enacted the Excellent Care for All Act in 2010 in an attempt to provide a provincewide focus on quality improvement.¹ Under this legislation, interdisciplinary health care organizations are required to submit quality improvement plans annually to Health Quality Ontario (HQO), a government agency that serves as the province’s advisor on health care quality. These quality improvement plans include an organization’s quality-of-care goals, performance indicators, improvement initiatives, and performance targets.² In general, the quality improvement plans aim to encourage focused quality improvement efforts on specific priority areas identified by HQO in collaboration with health care professionals and patients.

There are 3 components of a quality improvement plan. First, the work plan includes performance indicators and information on an organization’s current performance, target performance, and planned improvement initiatives for each indicator.³ (HQO recommends the inclusion of a predefined set of priority indicators, but organizations can, in addition to the predefined set, include other indicators.) Second, the narrative report provides contextual information such as an organization’s quality improvement commitments and goals; challenges to achieving the quality improvement goals and risk mitigation strategies; staff, leadership, and patient engagement; and leadership accountability. Third, the progress report describes the progress an organization has made toward improving performance for selected indicators over the past year. Once developed, senior leadership, the board of directors, and the quality committee (if applicable) are required to approve the quality improvement plan to demonstrate shared accountabilities and responsibilities. Quality improvement plans submitted to HQO annually are made publicly available to encourage shared learning among health care organizations.⁴

The aim of this study was to explore primary care administrators’ (ie, individuals who plan, coordinate, and oversee the functions of a primary care organization and staff) perceptions of the quality improvement plans, and barriers to and facilitators of using quality improvement plans as tools for improving the quality of primary care in Ontario. Insights were sought to inform the development of system-level strategies to enable quality improvement in primary care.

METHODS

Setting

In the primary care sector, organizations such as family health teams (FHTs) and community health centres (CHCs) have been submitting quality improvement plans since 2013; in 2014-2015, 185 FHTs and 75 CHCs submitted quality improvement plans to HQO.^5,6 These primary care organizations provide care to more than 3 million Ontarians.

Family health teams consist of interprofessional teams of health care providers—typically family physicians, nurse practitioners, nurses, pharmacists, social workers, dietitians, and other professionals—who work together to provide primary health care programs and services that are tailored for the community they serve.⁷ Most FHTs are governed by a provider-based board of directors. Physicians are affiliated with the FHT through an association agreement with their family health network, family health organization, or rural and northern physician group.

Community health centres also consist of interprofessional teams of health care providers but are typically governed by a community-based board of directors, care for a population that has high health care needs, and have an expanded scope of health promotion, disease prevention, and community development services.^8,9 Health care professionals, including physicians, are typically salaried employees.

Study design

A qualitative descriptive study was conducted.^10-12 This approach stays close to the surface of participants’ words and is particularly useful for examining practice- and policy-relevant phenomena. Compared with other qualitative methods, such as phenomenology, grounded theory, or ethnography, qualitative description is low inference, “offers a comprehensive summary of an event in the everyday term of those events … and researchers conducting such studies are the least encumbered by preexisting theoretical and philosophical commitments.”¹⁰

Eligibility and recruitment

Quality improvement plans are submitted electronically by Ontario primary care organizations to HQO. Quality improvement plans submitted in 2014 and 2015 by FHTs and CHCs were collected from HQO for the study. Based on the submitted quality improvement plans, FHTs and CHCs that reported an increase, decline, or no progress from 2013-2014 to 2014-2015 on 2 indicators were identified: percentage of patients who reported seeing their primary care provider on the same or next day when needed and percentage of patients who saw their primary care provider within 7 days after hospital discharge. These indicators were selected based on data completeness. Data for the indicator assessing same or next day appointments with a primary care provider were derived from self-reported survey data. Each organization administered a patient satisfaction survey, analyzed the data, and reported the results on their quality improvement plan. The indicator assessing posthospital primary care visits was derived from administrative data (ie, Discharge Abstract Database, Claims History Database, Client Agency Program Enrolment [Ontario Population Health Index of Databases], Corporate Provider Database). Included patients were rostered to an Ontario physician in a primary care practice model at the time of hospital discharge. Follow-up was restricted to services provided by a family physician, general practitioner, pediatrician, or geriatrician in the practice group the patient was registered with. Organizations accessed their data via the ministry’s Health Data Branch Web Portal and reported the result on their quality improvement plan.

Participants (eg, executive directors) who were involved with developing and implementing quality improvement plans were selected from these organizations. These participants and their contact information were identified through an online search and were recruited to the study via e-mail. Maximum variation sampling (based on performance, geography, and type of organization) was used to ensure participants with different levels of success in improving performance on the 2 indicators were interviewed.¹³ The 2 indicators were only used to identify and select participants, and did not influence the questions they were asked during the interview. Interviews were conducted until data saturation was attained—the point at which data collected did not contribute new information on the impact of quality improvement plans and the barriers to and facilitators of using quality improvement plans as a tool for improving the quality of primary care in Ontario.

Data collection

Semistructured interviews were conducted by a single interviewer (K.T.).¹⁴ The interview guide (available from CFPlus*) started with defining the study context and questions. Open-ended questions were then asked about the provincially mandated quality improvement plans in Ontario, which were followed by probing questions to pursue areas of interest. The questions in the interview guide were informed by a conceptual model of structural, organizational, individual, and innovation-related factors that influence the success of improvement initiatives (available from CFPlus*). The conceptual model was developed using theories of behaviour change and existing frameworks that describe factors that influence implementation and outcomes. It describes what and how structural-, organizational-, individual-, and innovation-level factors can lead to improved outcomes and quality of care. Structural-level factors include features of the external context or environment, organizational-level factors include features of the implementing organization, individual-level factors include characteristics of individuals involved with implementation, and innovation-level factors include features of an intervention that may affect implementation. All interviews were audiotaped and transcribed. The study received ethics approval from the University of Toronto Research Ethics Board.

Data analysis

To analyze data, a framework was developed to guide coding using constructs in the conceptual model, which was identified a priori to the analysis. The generated framework was then analyzed inductively to generate themes related to structural, organizational, individual, and innovation-related factors that influence the success of improvement initiatives. QSR NVivo software was used to manage the data.¹⁵ The principal author (K.T.) coded interview transcripts. Codes and associated interview extracts were collated into themes and subthemes, and were reviewed by the principal author to ensure they had internal homogeneity and external heterogeneity.¹³ All authors reviewed the conceptual model, themes, and subthemes.

FINDINGS

Eleven primary care administrators (ie, executive directors, directors of clinical services, office administrators) at 7 FHTs and 4 CHCs in Ontario participated. These individuals were generally responsible for planning, coordinating, and overseeing the functions of their primary care organizations and the staff who worked there, and were directly involved in developing quality improvement plans. Of the 11 participants, 5 were from organizations that reported an improvement on the 2 indicators. All interviews occurred via telephone and were conducted between October and December 2015. The views of participants who worked at FHTs were not notably different from the views of participants who worked at CHCs.

Findings have been organized into 3 interrelated themes: perceived impact of quality improvement plans, success factors, and challenges to improving quality of primary care. Each theme includes several subthemes.

Theme 1: perceived impact of quality improvement plans

Participants described how quality improvement plans led to 2 benefits: increased awareness of and focus on quality improvement, and implementation of strategies to improve quality of care.

Increased awareness of and focus on quality improvement. Most participants believed that quality improvement plans led to an increased awareness of performance measurement and quality improvement. Some participants also believed that quality improvement plans provided an overall focus on areas for improving practice and provided a structure for reporting on performance.

One participant (participant 6, executive director, FHT) stated, “I think there is a lot of benefit to [quality improvement plans]. For one thing it makes the staff aware that we need to focus on quality improvement all the time.” Another participant described how quality improvement plans increased awareness of quality improvement among the clinical and administrative staff, which has led to increased efforts to improve care:

I’d say [quality improvement plans] had been a little bit of … a driving force. I wouldn’t say they’ve had a huge impact just because at a community health centre [we are already doing] a lot of the things … it does make the providers more aware and … the front desk staff, like the medical secretary and receptionists who are booking appointments for example. It’s more in the forefront around them trying to get people in around 7 days of discharge from hospital. (Participant 2, director of clinical services, CHC)

Strategies to improve quality of primary care. Some participants described quality improvement plans, as well as the provincewide focus on improving care, as a driving force for implementing initiatives to improve quality of care at their primary care organization. For organizations that improved access to timely posthospital follow-up visits, the main strategy used was building partnerships with hospitals and patients. These partnerships helped ensure primary care organizations were informed when patients were discharged from the hospital. Several mechanisms were used to get timely discharge information: electronic notifications from hospitals, patient education letters, and primary care providers (ie, nurses, nurse practitioners) who actively engaged with hospitals to get discharge information and arranged postdischarge follow-up appointments for patients. To improve access to same- or next-day appointments, when needed, one organization increased the number of same-day appointments for urgent issues. These improvement efforts were driven in part or wholly by the quality improvement plans.

Theme 2: success factors

Participants described 4 factors that contributed to the success of quality improvement plans. These factors relate to attributes of the quality improvement plans (adaptability and compatibility) and contextual factors (leadership and organizational culture).

Adaptability. Participants appreciated that quality improvement plans were adaptable to their local practice. They liked the ability to include indicators that were important to their staff and organization. One participant stated the following:

I like that aspect of [the quality improvement plan] that these are the things [Health Quality Ontario] want answered, but these are the things [primary care organizations] can choose to answer. We picked up one extra thing this year to sort of work on but it gives us a bit more flexibility. (Participant 1, executive director, FHT)

Another participant (participant 5, executive director, CHC) said, “We had the opportunity to include things that we wanted to do, which was great … we did include some that were relevant.”

Compatibility with work flow. Some participants described that quality improvement plans were compatible with their existing work flow because they aligned with their organization’s efforts to improve quality of care. As one participant describes below, what changed was not their existing culture of quality but their approach to measurement:

We’ve always measured quality things. We’ve always had indicators for what we wanted to improve on. So the fact that we’re mandated now [to use quality improvement plans] didn’t change the culture here. It just maybe changed what we were measuring or forced us to measure more than we already were. The culture was already here, it was already part of what we believed in and thought about. (Participant 2, director of clinical services, CHC)

Leadership. Provincial leadership was described as an important factor for focusing attention on quality of care. One participant believed that without quality improvement plans, “nothing happens.” Another participant described how the Excellent Care for All Act increased attention on quality of care:

I think the legislation in Ontario has been helping at keeping quality front of the agenda and I would say that the plan itself as a tool for implementing the legislation … has been relatively useful and I think there’s been good improvements made each year. (Participant 9, executive director, CHC)

Participants also described that being accountable to the board of directors for quality improvement work and having quality improvement committees helped to increase use of the quality improvement plans.

Organizational culture. Some participants believed that quality improvement plans, alone or in tandem with other initiatives to improve quality of care, have “created an environment where we reflect on things.” For example, the quality improvement plans helped shift one organization’s culture from “physicians versus family health team” to a culture where physicians and staff work together on quality improvement:

I think [our culture] was a little bit more separate. I think that was why it was so daunting the first year we collected this data. Now it’s more, “Let’s work together to figure out what else we can collect.” It’s changed the culture from the physicians versus the family health team to “OK, we have to report this; let’s figure out ways of working around this together” and it’s better. It’s taken a bit to get there. It’s more collaborative. There’s a little bit more engagement of looking at how we can do things better, which there wasn’t any of that at all. Before it was just “People should be grateful that we’re here and accept what we give them.” (Participant 1, executive director, FHT)

Theme 3: challenges to improving quality of primary care

All participants described challenges affecting the success of quality improvement plans. Three challenges were described: data quality, staff and physician engagement and buy-in, and needed resources for measurement and quality improvement.

Data quality. Most participants believed that the data they received or aggregated themselves for the quality improvement plan indicators did not represent the quality of care provided at their organization. This could be attributed to a lack of standardized data and lack of real-time data to assess performance.

Participants described how data recorded in electronic medical records lacked standardization, which contributed to poor data quality. A participant described the lack of standardization, which negatively affects indicator findings:

[Electronic medical records] are a disaster; there’s no standardization, there’s no drop down, there was none of that forethought when implementing these in the practice … people report things totally differently … we started pulling data on [the immunization indicator] and the numbers just didn’t look right. And again it has to do with the way the data is being recorded at the user level. (Participant 11, executive director, FHT)

Participants also described the lack of real-time data, which made it difficult to monitor current performance and assess the impact of improvement initiatives. For example, data for some indicators were taken from administrative databases. A participant described how some indicator data were old and the frustration it causes:

A lot of the quality improvement plan data is old, which is really frustrating. It comes from other sources. It’s 2 years old. A lot of stuff has changed in 2 years, so it’s frustrating because you know you’ve done better but the results aren’t showing because it’s lagging. (Participant 11, executive director, FHT)

Staff and physician engagement and buy-in. Participants believed that quality improvement plan indicators did not accurately reflect the full range of their efforts to improve quality of care, which affected buy-in. For example, the indicator assessing posthospital follow-up visits used billing codes, which only capture visits to a physician. Participants described how telephone consultations or home visits with nurses or allied health professionals and patient choice were not captured in the data.

Participants also had differing opinions on what was important to measure. Some participants believed that the indicators were not meaningful to their organization and, if it was up to them, they would focus their efforts on other areas of quality. Others expressed the need for quality improvement plans that were tailored to the patient population they served (eg, rural communities, mental health). Quality improvement plans were generally viewed as a “one size fits all” tool, despite the fact that they were designed to be flexible.

In addition, some participants expressed that quality improvement plans were a top-down approach imposed by the ministry or HQO to improve quality, which led to some physicians being resistant to improvement efforts since it was intruding on their professional autonomy. A participant stated that physicians at her FHT

were not pleased with the ministry looking at what they were doing … that was difficult because the first thing that they wanted us to measure in the quality improvement plan was how long does it take for patients to access their physicians. (Participant 1, executive director, FHT)

Some participants expressed that improvement was outside of their control. Several reasons were given such as practice improvement required collaborating with other health care sectors and were therefore viewed as more system oriented. Patient-driven factors were also described. For example, some patients cannot or do not want to visit their primary care provider within 7 days of hospital discharge. Finally, some participants mentioned that physicians were outside of the control of FHTs and CHCs, but many of the indicators were physician focused. One participant (participant 8, office administrator, FHT) stated, “I don’t believe [physicians] have a sense of ownership. It’s difficult to have ownership if you don’t come to the table, so as much as we invite and have been inviting their cooperation, we don’t necessarily get it.”

Many participants expressed having competing priorities (eg, patient care, administrative work) that affected their level of engagement in quality improvement work. One participant expressed how patient care comes first, which means quality improvement work often falls behind:

What we do find and struggle with is the capacity piece. So we have a team that meets and … we’ve tried hard to re-integrate it to everything we do but it becomes difficult when you’re a smaller team because there are only so many people. If you’re directly impacting patient care you … have to serve the patient first and foremost. So fitting it in sometime in a more timely fashion is a challenge on teams because you’re always trying to shift priorities. (Participant 10, executive director, FHT)

Finally, participants described a lack of incentive to use quality improvement plans since there were no mechanisms holding organizations accountable for their performance. For example, there were no “carrots” to encourage practice improvements or “sticks” to penalize organizations that did not meet their improvement goals. As such, some believed that quality improvement plans were just an exercise that needed to get done.

Needed resources to support measurement and quality improvement. Most participants commented on a lack of internal resources and external contributions of resources. Many participants described a lack of time and resources that could be dedicated to performance measurement and quality improvement work. As a result, work associated with quality improvement plans had to be done outside of normal work hours. Participants believed that the ministry did not provide a corresponding level of investment to support the additional responsibilities. Some acknowledged having access to a quality improvement decision support specialist—a position funded by the Ministry of Health and Long-Term Care to help organizations meet their quality improvement goals—but believed that their specialist had limited capacity because they were responsible for the quality improvement activities at multiple primary care organizations.

DISCUSSION

Provincially mandated quality improvement plans in primary care in Ontario seem to have raised awareness of and provided an overall focus on quality improvement, and may have contributed to primary care organizations implementing initiatives to address quality gaps, which has the potential to improve quality of care. However, participants did not believe that the use of quality improvement plans have led to substantial improvements in the quality of primary care at their organization. This can be partly attributed to multiple challenges primary care organizations face, such as poor data quality resulting in organizations being unable to assess their performance in an accurate or timely manner, lack of staff and physician engagement and buy-in, and lack of resources to support quality improvement work.

To our knowledge, this is the first study examining the perceived impact of provincially mandated quality improvement plans on the quality of primary care. Findings emphasize the need to meaningfully engage and collaborate with stakeholders in the design, development, and implementation of improvement initiatives to increase buy-in.^16,17 In addition, the use of theory (eg, to understand why an intervention may lead to a planned change) or frameworks (eg, to understand factors that influence implementation and outcomes) is a critical step when designing and planning interventions intended to change behaviour.^18,19 Theories and frameworks can be used to do a thorough assessment of potential barriers at the intervention, individual, organization, and system levels so that strategies can be put in place to overcome these barriers and increase the chances of bringing about the desired change. In the case of quality improvement plans in Ontario, multifaceted strategies need to be co-developed by the ministry, HQO, and primary care organizations to improve data and measurement capabilities, improve staff and physician engagement and buy-in, and build capacity for quality improvement in primary care across the province.

The study findings are consistent with previous literature on quality improvement in primary care in Ontario. A study of another performance measurement effort in Ontario, the Association of Family Health Teams of Ontario’s Data-2-Decisions program to support the efforts of FHTs to measure and improve the quality of primary care, found that Data-2-Decisions did not lead to the desired quality improvement activity and points to the need to link measurement with action to change processes of care.²⁰ Similar barriers to improving the quality of primary care were described, including concerns with data validity, resource constraints, and the relationship between physicians and the rest of the primary care team.²⁰ Wagner et al describe the need to understand the motivations of primary care teams to participate in data-driven quality improvement and leverage their motivations to promote participation and engagement in performance measurement and quality improvement activities.²¹

Study findings support the notion that quality improvement plans are not “magic bullets” to improving quality of care. In reality, numerous factors (eg, related to the intervention, individual, organization, and political environment) interact to create an environment that enables or impedes the ability of quality improvement plans to drive improvements in patient care.^22-27 Often, quality improvement literature only describes the characteristics of an intervention, implementation process, and outcomes.²⁸ The findings presented here are important because they go a step further by offering explanations for why quality improvement plans may succeed or fail at improving the quality of primary care, which is rarely described in quality improvement literature.

Limitations

This study has several limitations. Only one individual per organization was interviewed, which may not be reflective of the views of others across the organization. Only the principal author coded the interview transcripts. However, all authors reviewed and agreed with the conceptual model that guided the analysis and reviewed the themes. The data are almost 6 years old and represent a snapshot in time. The sample size for the study may, arguably, be small. However, participants were interviewed to the point of data saturation, so it is unlikely that new themes would have been identified. Finally, the generalizability of the results to other jurisdictions may be affected by 2 factors: quality improvement plans in Ontario may be different than mandated plans elsewhere, and primary care capacity to enact quality improvement plans in Ontario may be different elsewhere.

Conclusion

Provincially mandated quality improvement plans in primary care may have increased awareness of and focused attention on the need for high-quality patient care, but participants expressed that the use of quality improvement plans have not yet led to substantial improvements in the quality of care provided at their organization. The lack of perceived improvements is likely the result of multifaceted and complex challenges primary care organizations face when trying to improve patient care. To effect positive change, organization- and health system–level efforts are needed to improve measurement capabilities, improve staff and physician engagement, and increase capacity for quality improvement among organizations.

Editor’s key points

• ▸ Provincially mandated quality improvement plans in primary care in Ontario seem to have raised awareness of and provided an overall focus on quality improvement, and may have contributed to primary care organizations implementing initiatives to address quality gaps. However, participants did not believe that the use of quality improvement plans have led to substantial improvements in the quality of primary care at their organizations.

• ▸ Multifaceted strategies need to be co-developed with primary care organizations to improve data and measurement capabilities, improve staff and physician engagement and buy-in, and build capacity for quality improvement in primary care across the province.

• ▸ The study findings support the notion that quality improvement plans are not “magic bullets” to improving quality of care. The findings presented here are important because they go a step further by offering explanations for why quality improvement plans may succeed or fail, which is rarely described in quality improvement literature.

Points de repère du rédacteur

• ▸ Les plans d’amélioration de la qualité dans les soins primaires, imposés par la province de l’Ontario, semblent avoir augmenté la sensibilisation et l’attention globale à l’endroit de l’amélioration de la qualité, et pourraient avoir contribué à la mise en place d’initiatives par les organisations de soins primaires pour combler les lacunes. Toutefois, les participants ne croyaient pas que l’utilisation de plans d’amélioration de la qualité avait entraîné des améliorations considérables dans la qualité des soins primaires dans leurs organisations.

• ▸ Il faut élaborer, conjointement avec les organisations de soins primaires, des stratégies à multiples facettes pour améliorer les données et les capacités de mesure, accroître la mobilisation et l’adhésion du personnel et des médecins, et édifier les capacités d’amélioration de la qualité dans les soins primaires partout dans la province.

• ▸ Les constatations de l’étude corroborent la notion selon laquelle les plans d’amélioration de la qualité ne sont pas une « solution magique » pour améliorer la qualité des soins. Les observations présentées ici revêtent de l’importance, parce qu’elles font un pas de plus en offrant des explications des raisons pour lesquelles les plans d’amélioration de la qualité peuvent réussir ou échouer, qui sont rarement décrites dans la documentation à ce sujet.