Assessing Continuous Quality Improvement in Public Health: Adapting Lessons from Healthcare

Abstract

Context:

Evidence of the effect of continuous quality improvement (CQI) in public health and valid tools to judge that such effects are not fully formed.

Objective:

The objective was to adapt and apply Shortell et al.'s (1998) four dimensions of CQI in an examination of a public health accountability and performance management initiative in Ontario, Canada.

Methods:

In total, 24 semi-structured, in-depth interviews were conducted with informants from public health units and the Ministry of Health and Long-Term Care. A web survey of public health managers in the province was also carried out.

Results:

A mix of facilitators and barriers was identified. Leadership and organizational cultures, conducive to CQI success were evident. However, limitations in performance measurement and managerial discretion were key barriers.

Conclusion:

The four dimensions of CQI provided insight into both facilitators and barriers of CQI adoption in public health. Future research should compare the outcomes of public health CQI initiatives to the framework's stated facilitators and barriers.

Introduction

This study examines the implementation of a public health accountability and performance management system featuring declared principles of continuous quality improvement (CQI) in Ontario, Canada. CQI is an approach to the management and improvement of organizational services and processes (Dilley et al. 2012; Nicolucci et al. 2010; Radawski 1999). The approach stands in contrast to quality control and assurance by virtue of its focus on identifying opportunities to improve work processes as opposed to identifying individualized problems and maintaining a status quo (Dever 1997). CQI relies heavily on performance measurement and analysis, as well as on the involvement of leadership and front-line staff in decision-making processes (Kosseff 1992; McLaughlin 1987; Radawski 1999).

The adoption of quality improvement approaches such as CQI in public health has been a recent and popular development (Capacity Review Committee 2006; Corso et al. 2010; Dilley et al. 2012). Despite this phenomenon, there exists a limited body of empirical evidence on the impact of quality improvement approaches in public health settings (Corso et al. 2010; Dilley et al. 2012; McLees et al. 2014; Riley et al. 2012). Moreover, valid and reliable frameworks for assessing the integrity and impact of such systems in public health are still emerging. In contrast, development of CQI in healthcare settings has been much more extensive, dating back to the late 1980s (Chinnaiyan et al. 2012; Radawski 1999; Rex et al. 2002).

Adapting Healthcare Quality Improvement Knowledge for Public Health

This study uses Shortell et al.'s (1998) four dimensions of CQI as an analytical framework for assessing a public health quality improvement initiative in Ontario (Figure 1). The four dimensions of CQI represent an assessment framework derived from systematic reviews of empirical healthcare research. In addition, Shortell et al.'s earlier research on the cultures of high-performing organizations is used to augment the cultural dimension of the adapted framework (Shortell et al. 1995). For instance, developmental cultures featuring an emphasis on risk-taking, innovation and change, as well as group cultures with strong teamwork and participation, found the greatest success in supporting CQI initiatives. Hierarchical and rational cultures that stress bureaucratic norms and narrow definitions of achievement were found to act as barriers.

Figure 1.

The four dimensions of continuous quality improvement (CQI)

Within clinical health research fields, the four dimensions of CQI have received empirical validation (Bennett and Crane 2001; Forsner et al. 2008; Solomons and Spross 2011). One example includes Forsner et al.'s (2008) controlled study of evidence-based practice in Swedish psychiatric care. The investigators examined the implementation of clinical guidelines and found that in the test group, in which the four dimensions of CQI were applied, the reported guideline compliance was significantly greater (p < 0.001) than in the control group.

Many of the key factors presented in the four-dimensions framework overlap with those emphasized in the public health quality improvement literature. For instance, in their qualitative study of 51 quality improvement initiatives in various public health departments in the US, Riley et al. emphasized leadership and appropriate performance measures, which are also highlighted in the strategic and technical dimensions of CQI (Riley et al. 2012). Resource inadequacy was a key barrier in McLees et al.'s (2014) study of 74 public health agencies involved with the National Public Health Improvement Initiative in the US, as it is in the strategic dimension of Shortell et al.'s framework. The importance of training and education in quality-improvement concepts and techniques is also stressed in both the public health literature and the technical dimension (Corso et al. 2010). Knowledge transfer and exchange, supportive organizational cultures and the influence of implementers in decision-making outlined in the structural, cultural and strategic dimensions were factors that did not appear to have extensive profiles in the public health literature.

Research Context

The public health system in Ontario features several key stakeholders, including the Ministry of Health and Long-Term Care (MOHLTC), boards of health and local public health units. The Ministry provides provincial stewardship and 75% of core funding for the public health system and is also charged with upholding key legislation. Boards of health are municipal and regional public health governing bodies that are responsible for overseeing their corresponding public health units and providing them with the remaining portion of core funding. Public health units are the agencies that deliver programs and services in their respective jurisdictions.

Following the 2003 outbreak of Severe Acute Respiratory Syndrome (SARS) in Ontario, a major reform of the system was initiated by the Minister of Health (Smitherman 2004). This reform included the declarative adoption of CQI as a means of pursuing performance improvement (Capacity Review Committee 2006; Law et al. 2013; MOHLTC 2008, 2011). CQI adoption has taken the form of a system of accountability and performance management, currently undergoing implementation across 36 municipal and regional public health jurisdictions. The system is composed of (1) the Ontario Public Health Standards (OPHS), which outline the program and service requirements for boards of health and public health units, as well as broad goals and outcomes across each area of public health; (2) accountability agreements between the Ministry and boards of health and their public health units that establish specific performance indicators and targets related to areas of the OPHS; (3) organizational standards that articulate management and governance requirements for boards of health and public health units; and (4) reporting requirements for the collection and analysis of performance measurement information (MOHLTC 2008, 2011, 2013).

Methods

A mixed-methods approach consisting of key informant interviews and a web survey was used to assess Ontario's public health accountability and performance management initiative. All data collection and analysis were conducted by the principal author with university ethics approval and editorial feedback from co-authors. This study used many elements of a case study approach, such as interview and survey methods and triangulating analysis, which have been used extensively in the field of implementation research (Long and Franklin 2004; McDermott 2004; Mischen 2006). Research conforming to case study characteristics has also been used to investigate public accountability and performance management (Christensen and Lægreid 2014; Hildebrand and McDavid 2011).

Key informant interviews were conducted in three public health units (sites A, B and C). Sample selection was conducted to reflect diverse implementation contexts characterized by both rural and urban service environments, as well as municipal and regional governance. In total, 20 semi-structured key informant interviews of ~1 hour in length were conducted. All interviews were tape-recorded and professionally transcribed in full. Public health unit informants included executive, management and specialists in various areas of public health, such as chronic and infectious disease prevention and control. These groups of individuals represent the primary implementers of the province's public health accountability and performance management intervention.

Four separate interviews with representatives of the MOHLTC were also conducted during the same period of data collection. These interviews also followed a semi-structured approach and included individuals directly involved with the development of the province's quality improvement initiative.

Interview questions for both public health unit and Ministry informants included specific and broad items relating to facilitators and barriers within the four dimensions of CQI. For example, Ministry and public health informants were asked to choose characteristics of Shortell et al.'s (1995) organizational cultures typology (i.e., teamwork, risk-taking, bureaucratic, efficiency-focused) that best reflected their work environment. Broader items included questions asking informants to independently identify what conditions or factors were critical to the success of implementing Ontario's system of accountability and performance management.

In addition to key informant interviews, a web survey of public health managers was conducted. Targeting all public health managers in each of the province's 36 health units, recruitment involved contacting each senior executive to seek approval and access to their organizations. In total, 12 public health units agreed to participate and provided contact lists of public health managers; 97 surveys were distributed; 53 questionnaires were returned, providing a response rate of 54.6%. This sample, while only covering one-third of all public health units, represented a near-equivalent distribution of rural, mixed rural and urban, and urban jurisdictions. Survey questions, for example, asked about manager discretion, relating to stakeholder decision-making in the strategic dimension; resistance to the intervention, relating to barriers in the cultural dimension; familiarity with components of the initiative, as well as sentiment regarding performance measurement pertinent to the technical dimension; and prospective thoughts on the use (and usefulness) of collected information for performance management and quality improvement.

Directed content analysis was applied to qualitative data by using an initial coding frame informed by pre-existing empirical and theoretical literature (Hickey and Kipping 1996; Hsieh and Shannon 2005; Potter and Levine-Donnerstein 1999). Strong, anomalous themes were then coded separately. Established codes were then matched with facilitators and barriers of Shortell et al.'s (1998) four dimensions of CQI and analyzed. Quantitative data collected using keysurvey.com were recoded for descriptive and bivariate analyses using SPSS. Two-sided Fisher's exact tests (p ? 0.05) assessed association because of the small survey sample (Daya 2002). Findings from the survey supplement the qualitative data, and all presented findings did not feature missing data (n = 53).

Findings

The findings in this study are presented across strategic, cultural, technical and structural dimensions and focus primarily on the facilitators and barriers in the four-dimensions framework. Overall, evidence of both facilitators and barriers in each dimensional category related to Ontario's system of public health accountability and performance management was apparent.

Strategic dimension

The strategic dimension emphasizes the importance of leadership, communication and inclusion of all stakeholders in decision-making. Analysis of interview data found statements of strong leadership expressed by each public health unit. Local-level leadership in quality improvement focused mainly on outcomes in priority populations, such as immigrants from countries with endemic infectious diseases. Ministry informants identified leadership as a key driver of implementation efforts and acknowledged its strength within public health units, who they felt shared their interest in showing high performance. A Site-B informant confirmed this leadership sentiment:

“Well, we have very strong leadership values of teamwork and participation and participatory management in most of our program areas. I think we are very strong that way.” – Site-B informant

Communication relating to the initiative was evident from interview findings identifying various forums for the development and conveyance of its elements. Several public health unit informants noted that the province's new system of accountability and performance management had prompted both internal and external dialogue, which has since increased their understanding of performance objectives and quality improvement more broadly. Site-A stood out as a particularly strong example of this:

“I think that changing conversations has actually motivated people, not just here in the health unit but even as I talk to people across the province. People like the fact that we are being asked to think about these questions and like the fact that we are going to be held more accountable for actually making a difference.”

– Site-A informant

Similarly, a large proportion of survey respondents reported moderate or great familiarity with many components of the intervention, including accountability agreements (96%), performance targets (98%) and reporting requirements (94%).

Implementer inclusion in decision-making was mixed. Although many public health unit informants cited participation in committees and working groups related to the CQI initiative, their influence over final decisions varied. In some cases, such as human papillomavirus (HPV) vaccination, public health agents were able to negotiate “more realistic” performance targets.¹ In other instances, local informants noted that the Ministry took a hard line in making decisions despite concerns voiced by the field. For example, when a prescriptive OPHS protocol for tuberculosis follow-up was challenged because of evidence of alternative best-practice, requests to change the protocol were denied by provincial decision-makers. One Ministry informant corroborated this dynamic by stating their interest in the input provided by the field, but the decisions ultimately rested with those holding authority over legislation:

So, yes, [consultation] is to enable conversations within a forum that in a sense the majority of the practitioners and the province have agreed to talk about. It's supposedly a partnership. The province always has the upper hand. (laughing) He who controls legislation has the upper hand.” – Ministry informant

Informants in each of the three public health unit interview sites raised concerns over the narrow timelines for achieving targets. One Ministry informant noted that many of the targets are set to 100%, matching with the OPHS, and that even public health units with low baselines would be expected to meet targets within the first two years of implementation.

Divergence between the Ministry and the field was expressed in terms of provincial and local health priorities. Some informants argued that targets set by the Ministry such as senior falls were not a priority in their jurisdiction or, generally, a major responsibility of public health because of small target populations and the many determinants outside of their control. Although many public health unit informants acquiesced to the province's quality improvement initiative, each of the local public health unit interview sites placed greater emphasis on internal systems of performance management to foster meaningful performance improvement. For instance, one Site-A informant stated:

“At this point I feel more confident in our organization's capacity to demonstrate success in performance management than I do with the two indicators my team has been given within the accountability agreement system from the Ministry of Health and Long-Term Care.” – Site-A informant

Despite the implied and explicitly stated opportunity cost created by misalignment in local and provincial priorities apparent in interview findings, 74% of survey respondents disagreed or strongly disagreed that an emphasis on provincial performance measurement and target achievement would interfere with the quality of program and service provision at a local public health level.

Resource inadequacy was often referenced in relation to the cost neutrality of the intervention and the current public health funding model, more broadly. While public health unit informants highlighted the quality of their agencies' human resources, some did not consider general resourcing to be adequate for achieving all targets – a phenomenon that was reflected by nearly one-third of surveyed public health managers. Issues of increased burden on public health units to show compliance with provincial targets and fulfill local priorities were, in some cases, compounded by rapidly expanding local populations that the current public health funding model does not compensate for. For example, one Site-B informant explained:

“I think that both financial and human resources, I think for most if not all boards of health [our] reach is beyond our grasp … [our] population increases five to ten thousand a year. Basically I've been getting base budget increases for the last few years. In other words very few if any new staff to service a population even over the last four years that would be in the order of twenty to forty thousand additional people.” – Site-B informant

Ministry informants acknowledged the need for greater equity in the public health funding model, although some were not convinced that public health performance improvement required additional funding, but rather greater efficiency. Other barriers such as work overload did not have a strong profile in the data, although some public health unit and Ministry informants speculated that smaller, rural health units may struggle with performance expectations related to intensive analytical tasks such as population health assessment.

Cultural dimension

Facilitators of the cultural dimension are distinguished by openness, collaboration, teamwork and learning. At the local level, all three public health unit interview sites exhibited at least some of the characteristics of developmental and/or group culture. Site-A exhibited many characteristics of group culture, such as teamwork and participation. Organizational hierarchy appeared fairly flat, and even front-line workers were said to be involved in program decision-making, collective priority-setting and performance monitoring. One Site-A informant noted:

“Certainly I think we prided ourselves on teamwork and participatory management styles and participation of front-line staff into decision-making where that makes sense.” – Site-A informant

Site-B appeared to be an equally distributed mix of developmental, group and rational cultural types – emphasizing efficiency and achievement of OPHS requirements. Group culture was apparent in reference to the interdisciplinary team-based approach to program and service provision. Leaders also regarded teamwork as an important value of their culture, as illustrated by instances of participative management in various program areas. Developmental culture emerged in the context of the health unit's internal, evidence-based approach to CQI planning, which allowed for informed innovation and risk-taking. A Site-B informant expanded by stating:

“… there has been a very strong undercurrent in my organization … that your programming is evidence-based and you have a method for reviewing it and each time trying to learn more about how it went and improve it. It's a continuous cycle of implementation, reflection, evaluation, and review and kind of revision. So there is constant introduction of innovation as well as fine-tuning things as they go.” – Site-B informant

Site-C presented a dominant developmental culture. Risk-taking and innovation were often regarded as very important aspects of the organization's culture. These aspects of developmental culture were contextualized in terms of evidence-informed decision-making, which was paradoxically argued to reduce risk at the same time. Risk-taking in the development of strategic plans and priorities and examples of innovative programming were highlighted as proof of the health unit's commitment to a developmental culture. An example of one Site-C informant reflecting on the health unit's organizational culture explained that:

“… evidence informed decision-making is a large component. It's one of the strategic priorities in our health unit and so really having that … engaging in processes of informed innovation certainly informs decision-making and out of that what are the risks that we are taking to do things differently than other health units based on the evidence that we have found.” – Site-C informant

Resistance to change, unrewarded achievement, and hierarchical and rational organizational cultures are regard as barriers to the cultural dimension. In this regard, there was limited evidence of an approach for rewarding achievement and good performance related directly to the province's CQI initiative. However, some public health unit interviewees argued that celebrating achievement of targets was important – something that their health units did internally when goals were achieved or improved upon. Site-A provided an example of this:

“I think setting targets and celebrating the reaching of the targets is the other part. Part of our plan will be not just setting goals but also celebrating the achievement of the goals … What we look at when we set out goals for staff and within the organization, knowing that we are not going to achieve every goal every time but celebrate our achievements and keep us moving forward.” – Site-A informant

Moreover, there was some uncertainty about the level of support for facilitating factors and the presence of barriers such as hierarchical cultural norms. On the issue of whether the system was primarily intended to promote learning (a key characteristic of CQI), those surveyed in the area of chronic disease prevention were significantly more uncertain than respondents from other areas (p < 0.02). Likewise, 84.3% of the survey sample agreed or strongly agreed that the initiative was primarily concerned with maintaining compliance with public health practice and performance expectations – resembling a quality assurance orientation. In addition, respondents in the area of emergency preparedness were more likely to disagree (p < 0.02) with the statement that data generated from the provincial initiative would be used to improve performance.

Technical dimension

Training opportunities and the quality and availability of data are the primary facilitators in the technical dimension. In Ontario, training in quality improvement, and CQI specifically, manifested mainly at the local level, with public health units providing instruction to staff on strategic planning. Guidance in program and service provision was evident through provincial OPHS protocols, but these materials did not relate specifically to quality improvement training. A Site-A informant reflected on this gap:

“I'm not aware of any kind of … the how stuff that's come from the Ministry other than just … okay your targets are now being established with an expectation we do something about them.” – Site-A informant

Gaps in training and data systems can be precursors to frustration and false starts, according to Shortell et al. (1998). Guidance from the Ministry on how public health units were to achieve performance targets or improve was limited. Moreover, some guidance materials, such as the previously mentioned tuberculosis protocol, were criticized by Site-C informants for not reflecting best available evidence and local expertise:

“So we had examples where we are absolutely convinced that we should vary the standards or not conform exactly with the [tuberculosis] protocol. This is the Ministry telling us how to practise public health where actually we know more about practising public health than they do … It always ends up the same way because their lawyers advise them to stick to the letter of the law. I don't know. Something to do with liability. This is not the best use of our resources.” – Site-C informant

Considerable concern with the quality of performance measurement information relating to the provincial initiative was raised by all parties. The choice of population health outcomes as measures of public health performance was identified as problematic because of externalities that made attributing public health outputs difficult. Some performance indicators were perceived as unreflective of public health performance by health units. For instance, the tobacco use indicator was highlighted as one such problematic measure:

“We only have one performance indicator that relates to chronic disease and that is the one about the number of youth who smoked a whole cigarette. I think it doesn't reflect in any way the work that we do but I understand the Ministry's need to show a tangible objective outcome and so we will do that and be happy with providing that information. I would say it has very minimal contribution to anyone understanding anything about what we do.” – Site-B informant

Ministry informants generally agreed that information systems needed to be improved and that this task was a difficult one. However, Ministry informants also noted that where evidence was weak, best-practice information was used in place of causal linkages between OPHS requirements and outcomes. One Ministry informant explained:

“So the real work is at the linkages between requirement, to short-term outcome, to medium-term, to long-term outcome … so wherever we made a link we found evidence to support that but where we couldn't, it was based on best practices and what was occurring in the field and the assumptions that were being made that had been integrated right at the beginning of the '98 standards all the way through.” – Ministry informant

Structural dimension

The structural dimension focuses on effective forums of communication for facilitating learning throughout an organization or system. In Ontario, the CQI initiative is supported by several communication forums, such as accountability agreement working groups, committees and monthly teleconferences amongst public health specialists, leadership and the Ministry. At a local level, several public health unit informants noted active lines of communication between themselves and other public health units pertaining to collaborative projects, research and other forms of knowledge production and exchange. In contrast, some Ministry informants stated that public health units do not typically work cooperatively or collaboratively because of jurisdictional protectionism. This divergence in perspectives was reflected in informants' testimony:

“The fact that none of them work cooperatively, the fact that there are turf wars and all that good stuff, I think is one of the challenges.” – Ministry informant

“I think there is a lot of similarities between health units. We talked a lot.” – Site-B informant

“We work really well with our partners so we can capitalize on limited resources and make the most of them so that again we can really accomplish the goals we set out for communities and make our communities healthier places to be. So we do a lot of collaborative work with other health units but also with our community partners as well in order to accomplish public health goals.” – Site-A informant

Within the structural dimension, the lack or limited use of communication mechanisms related to the quality improvement initiative fosters an inability to produce knowledge and diffuse it within systems. Ambiguity relating to how information would be fed back to public health units and used for quality improvement was apparent and highlighted by local informants:

“I don't know. I think that remains to be seen. I'm hoping it's more to be used in a combination with evidence to make ongoing improvements to public health programs and policies.” – Site-C informant

Ministry informants stated that performance information would allow for “discussion” with public health units. Some public health unit informants speculated that these discussions would include questions of what barriers to performance existed. One public health unit interviewee noted that performance information lacked the context to address why the results were the way they were. Meanwhile, several other informants argued that their public health unit would have to provide additional, unsolicited information to explain their performance achievement. One Site-B informant explained:

“So, in my earlier interview with you I described some of the vehicles that you can use and that I voluntarily send to the Ministry like our performance report, it's rare that I would get an acknowledgement, let alone do they read it. So I don't think the Ministry is all that interested in what we are doing apart from the information that we use to populate what I would call to be our financial reports.” – Site-B informant

In contrast to the provincial initiative, all public health unit interview sites described specific internal processes of quality improvement such as balanced scorecards, evaluation, reporting and strategic planning elements. Only 55% of survey respondents believed the province's system of accountability and performance management had the intent of providing learning opportunities and improving performance.

Discussion

This study shows a mix of facilitators and barriers to CQI best-practice in Ontario, according to Shortell et al.'s (1998) four-dimensions framework. Evidence of strong leadership interest and involvement in quality improvement at both local and provincial levels was clear. Strong developmental and/or group cultures were also evident at public health unit interview sites, which reflected leadership efforts to foster high performance and provided additional insight into their CQI capacity. The importance of senior and managerial leadership engagement cannot be over-emphasized, as previous reviews of public health quality improvement initiatives have shown (Dilley et al. 2012; Randolph et al. 2012). However, Ontario's quality improvement initiative also featured limitations placed on the meaningful influence of local leadership in decision-making by provincial stakeholders, which was reflected by misalignments in priorities, even though agreement on the principle of quality improvement was mutual. A part of this phenomenon may be because of the split emphasis that Ontario's system of accountability and performance management has between quality assurance and quality improvement. Assurance of legislative and service requirements promotes top-down decision-making and control, whereas a focus on improving outcomes requires local leadership and discretion. Similar misalignments were highlighted in the work of Degroff et al. (2010) who argued that many of the challenges to applying performance measurement to national public health programs in the US were due, in part, to the competing interests of quality improvement and public accountability (DeGroff et al. 2010).

In addition, the availability of indicators that accurately reflect performance continues to be one of the greatest constraining factors to CQI in public health settings, as many have already pointed out, and one that sets it apart from healthcare (Kahan and Goodstadt 1999; Scutchfield et al. 2009; Weir et al. 2009). Public health unit informants were adamant that performance targets indicated by population health outcomes, which are subject to numerous determinants outside of their control, were problematic. Given that CQI relies upon the quality of performance measurement information for informed decision-making, developing public health metrics that are more attributable to service outputs should be a priority.

Limitations

This study is limited by its small public health unit sample, which, although offers valuable insight into a nascent quality improvement process, ultimately, cannot represent the broader set of units. Also, while boards of health are acknowledged to be important stakeholders in Ontario's public health system, members were not included in this study because of unsuccessful recruitment. Furthermore, this study offers a snapshot of an initiative in a fluid environment and in its very early stages. Changes to the approach are expected, which have potential implications on the perceptions of informants. This also means that evidence of outcomes resulting from the presence of facilitators and barriers was beyond the scope of this study.

Conclusion

This study illustrates the applicability of Shortell et al.'s (1998) four dimensions of CQI as a framework for understanding public health quality improvement. The study also represents one of the first attempts to examine the implementation of a CQI initiative across a complex public health system using an empirically derived and validated framework from the healthcare field. Insight provided by the framework relating to facilitators and barriers of CQI implementation has largely confirmed disparate public health research on the topic (Corso et al. 2010; McLees et al. 2014; Riley et al. 2012; Shortell et al. 1998). This confirmation is a promising indicator that the framework may hold value as a tool for public health decision-makers developing and implementing CQI systems. Finally, future research should test the four-dimensions framework in other public health environments and, more importantly, examine the linkages between the framework's indicated outcomes and attributable facilitators and barriers.