The outcome competency framework for practitioners in infection prevention and control: use of the outcome logic model for evaluation

Abstract

Healthcare is delivered in a dynamic environment with frequent changes in populations, methods, equipment and settings. Infection prevention and control practitioners (IPCPs) must ensure that they are competent in addressing the challenges they face and are equipped to develop infection prevention and control (IPC) services in line with a changing world of healthcare provision. A multifaceted Framework was developed to assist IPCPs to enhance competence at an individual, team and organisational level to enable quality performance and improved quality of care. However, if these aspirations are to be met, it is vital that competency frameworks are fit for purpose or they risk being ignored. The aim of this unique study was to evaluate short and medium term outcomes as set out in the Outcome Logic Model to assist with the evaluation of the impact and success of the Framework. This study found that while the Framework is being used effectively in some areas, it is not being used as much or in the ways that were anticipated. The findings will enable future work on revision, communication and dissemination, and will provide intelligence to those initiating education and training in the utilisation of the competences.

Introduction

The World Health Organization (WHO), in a report on the impact of healthcare associated infections (HCAI), summarised the problem as follows: “The impact of healthcare associated infection implies prolonged hospital stay, long-term disability, increased resistance of micro-organisms to antimicrobials, a massive additional financial burden for health systems, high costs for patients and their families and excess deaths” (WHO, 2011, p. 3). Additionally, the WHO (2011) report also highlights that HCAI are a hidden problem that cut across all healthcare settings and are yet to be solved nationally and internationally. However, recent demonstrable improvements in significant HCAI (e.g. meticillin resistant Staphylococcus aureus bacteraemia and Clostridium difficile infection) across the UK indicate that even if the war is not won, significant battles are having an impact. Embedding high standards of infection prevention and control (IPC) practice is fundamental in improving the safety and quality of patient care. To achieve this, it is essential to have a competent and confident workforce to address the continuing and evolving challenges of reducing and controlling HCAI.

In 2011, the Infection Prevention Society (IPS), in partnership with the national health departments, Skills for Health, the Council of Deans and other stakeholders published the Outcome Competences Framework for Practitioners in Infection Prevention and Control (Infection Prevention Society, 2011) (hereafter called the Framework). The purpose of the Framework is to assist practitioners working in IPC to build competence throughout their career so that they can provide expert advice and leadership in the pursuit of improved quality of care for patients through the reduction in the burden of disease and illness caused by HCAI.

Since the publication of the Framework, national policy changes across the UK have resulted in mergers, reorganisation, downsizing and redesigning of workforces. These national policy changes are creating real challenges for healthcare organisations and practitioners. Despite this, the expectation is that quality care must be delivered in a more efficient and cost effective way and that standards must be improved (Walshe, 2010). The rationale for evaluating this Framework at this time is the belief that good performance reflects good-quality practice, and that comparing performance between providers and organisations will encourage better performance (Hughes, 2008). Additionally, since the Framework’s publication and advocation of usage, sufficient time has passed to assess its implementation and impact. The implementation of programmes or interventions takes a lot of effort and a systematic approach to ensure that professionals learn, adapt and sustain effective ways of working. To our knowledge no other competency framework of any description or set of statements has ever undergone such a rigorous evaluation.

Aim

The aim of this study was (1) to identify the scope of access to the Framework among practitioners, (2) to examine the perceived potential impact of the Framework, and (3) to explore the awareness, uptake and implementation of the Framework within the specialty of IPC.

Methodology

The Outcome Logic Model was used to provide a framework for evaluation. This model is an effective tool for plotting the path of a programme from creation through implementation to outcomes (Armstrong and Barsion, 2006). It provides a graphical and textual representation of how a specific programme is intended to work and links outcomes with the processes and assumptions underpinning a programme (Hayes et al, 2011). There are three major components to the Outcome Logic Model, which are developed in the context of a programme’s assumptions and external factors (Mederiros et al, 2005):

• Inputs (what resources were used by a programme?)

• Outputs (what was done (activities) and who was reached (participants)?)

• Outcomes (what are the results of the programme?)

The programme development group and stakeholders articulated the desired outcomes of the Framework to ensure the overall goals were clear and unambiguous (Haggard and Burnett, 2006). Using the Outcome Logic Model, Figure 1 provides a visual representation of the three major components of the logic models: inputs, outputs and outcomes.

Figure 1.

The Outcomes Logic Model for the Outcome Competences for Practitioners in IPC (IPS, 2011)

Inputs

As with all logic models, inputs are what was invested in order to develop the Framework, which included funding and human resources. The IPS Education and Professional Development Committee undertook the initial work in developing the Framework, sharing it with key stakeholders in the development stages. This led to a recognition that the work filled an important gap in efforts to improve patient safety and was followed by the establishment of a national steering group chaired by the Chief Nursing Officer for Scotland, which consisted of IPS members, representatives from all four United Kingdom (UK) government health departments, Skills for Health, Council of Deans and wider stakeholders. The project was funded by the Department of Health (England). Following seven months of meetings, collaboration with stakeholders and revision of draft work, the Framework was complete.

Outputs

The outputs of a logic model are categorised as activities and participants. The activities relate to what was undertaken after the completion of the Framework and the participants relate to the individuals who were reached by the programme. This enables the establishment of linkages between the situation and the outcomes of the programme. The outputs of the framework were detailed in the IPS and all four UK government communication strategies.

Outcomes

In order to determine the outcomes of a programme, evaluation questions and indicators must be set. While there is no one ‘right’ indicator for an outcome, Armstrong and Barsion (2006) state that an indicator needs to be credible, observable and measurable. As such, the evaluation plan should identify indicators relevant to the short, medium and long term outcomes.

The next stage of the evaluation cycle is to collect and analyse multiple data sources by using quantitative and/or qualitative methods. Data will then provide a purpose for the further development and improvement of the Framework (Kazi, 2003).

Research design

Methods

An electronic questionnaire was developed through Survey Monkey™ and distributed by email to practitioners, managers and academics working in IPC. The survey comprised 25 dichotomous, multiple choice and open-ended questions: eight demographic questions and 17 questions specifically related to the Framework. The full survey is available as an electronic file. The questionnaire was piloted for understanding and ease of use among ten academics and researchers. Minor changes were made following the pilot prior to the distribution of the questionnaire to the study sample.

Study sample

The study sample was recruited via the IPS member database (n=1,802) and wider infection prevention and control stakeholders throughout the UK. These included representatives from the Healthcare Infection Society (HIS), Royal College of Nursing (RCN), NHS Education for Scotland (NES), National Resource for Infection Control (NRIC) and infection control managers. An initial email was sent providing details about the study and a link to the questionnaire, with a further two reminders thereafter. All individuals with IPC roles and responsibilities were eligible for inclusion. The wider stakeholders were asked to forward the email to their IPC networks. The data were collected over a five-week period between September and October 2012.

Ethical considerations

Ethical approval was granted from the University of Dundee ethics committee. A full explanation of the study was provided to all participants on receiving the online questionnaire in the form of an email explanation and the attachment of a participant information leaflet. Completion of the questionnaire served as consent.

Data analysis

Socio-demographic data were calculated as frequencies (%), means, standard deviations, range, medians and interquartile range (IQR). Questionnaire data were calculated as frequencies (%). Chi-square and Fisher’s Exact tests (where appropriate) were performed to examine differences between categorical/nominal socio-demographic and/or questionnaire variables. In addition, independent group statistical analysis for two (Mann-Whitney U) or more groups (Kruskal-Wallis test) were performed to identify significant associations between ordinal or ratio-scale socio-demographic variables and questionnaire variables. IBM SPSS® 20.0 for Windows was used to aid statistical analyses. All tests were conducted with a two-tailed level of significance of p<0.05.

Results

Two hundred and ninety-four questionnaires were returned. Of these, two questionnaires were excluded as they contained no data and 50 were excluded because they contained socio-demographic data only. A total of 242 questionnaires were included in the study, giving a response rate of 13.42%.

Socio-demographic characteristics

Table 1 and Figures 2–5 provide summaries of the respondents’ socio-demographic characteristics. Respondents had been working in their current post for a median of five years (IQR 3–8), although they had been in a role with infection prevention responsibilities for a median of eight years (IQR 5–15).

Table 1.

Socio-demographic characteristics

Variable	Category	n	%
Age (years)	26–35	19	8
	36–45	82	34
	46–55	105	44
	56–65	32	13
	66+	1	0
Work rolea	Responsibility for 2 or more healthcare organisations	44	18
Job titlea	Nurse (other than director/manager)	181	75
	Doctor, microbiologist, other	41	17
	Director, manager	19	8
Bandb	5–6	53	25
	7–8	62	29
	>8	66	31
	Other	30	14
Place of residence	Scotland	15	6
	Northern Ireland	7	3
	Wales	7	3
	England	187	77
	Other	26	11
	Mean (SD)	Median (IQR)	Range
Time in current post (years)c	6.01 (4.99)	5.00 (3.00–8.00)	0.17–32.00
Time in a role with IP responsibilities (years)d	10.50 (7.26)	8.00 (5.00–15.00)	0.17–40.00

SD=standard deviation; IQR=interquartile range; IP=infection prevention

^an=241; ^bn=211: A ‘band’ is indicative of an individual’s job role (the higher the band = the higher the job role); ^cn=235; ^dn=236

Figure 2.

Respondents’ place of work (n=241)

Figure 3.

Job titles of respondents (n=241)

Figure 4.

Respondents’ qualifications (n=238)

Figure 5.

Respondents’ place of residence (n=242)

The majority of respondents indicated awareness of the existence of the Framework (90%, n=218), with 75% (n=182) reporting that they had read it. Of the 75% of participants who had read the Framework, 84% (n=162) stated that it was easy to understand, with 10% (n=20) stating that it was not. Only 45% (n=109) of the sample answered that they had received a hard copy of the Framework, predominantly from IPS as IPS members (95%, n=106). Of those who had not received a hard copy of the Framework, 71% (n=117) had accessed it electronically.

Nearly half the respondents (43%, n=99) reported that they were using the Framework in practice. When asked if there were perceived barriers to the use of the Framework, 30% (n=68) indicated there were. However, the majority felt that use of the Framework was beneficial (61%, n=140), with the remainder being unsure (36%, n=84). The main reason cited by participants who had not read the Framework was lack of time.

The participants who stated that it was not easy to understand (10%) said it was too long, overwhelming and repetitive in some places; there was too much jargon; they were confused about how to use it, especially in the absence of a self-assessment form; and there were too many professional conflicting priorities and time constraints. One participant also thought that the Framework was inapplicable to a new IPC and another stated they would have preferred specific competences related to novice, intermediate and expert practitioners.

In contrast, however, nearly half of the participants who were using the Framework stated that it enabled an enhanced and structured learning and development programme which assisted with career plans, gave recognition to the specialist role, helped practitioners keep up-to-date with current infection prevention issues, helped identify gaps in practice, gave assurance of good practice, enabled a standardised approach to development, improved standards and quality care, provided a national benchmark that recognised competence within this field, allowed practitioners’ roles to be defined and justified and enabled the alignment of the Framework with national Knowledge Skills Framework (KSF).

Approximately 58% (n=135) of the sample reported that their line manager was aware of the Framework; and 32% (n=73) and 30% (n=70) indicated that they had been encouraged to use the Framework by their managers or their peers, respectively. Individual comments revealed that some participants were not encouraged to use the Framework by their managers, although this was mainly due to the managers not being aware of its existence. Some managers were not in the field of IPC and lacked awareness of the need for practitioners to be autonomous in their own development. In contrast, those whose managers had encouraged practitioners to use the Framework stated that their manager had integrated it in the IPC team’s annual development programme, and had used it for performance management and staff appraisal and to develop induction programmes for new staff.

Of those who used the Framework in practice, 64% (n=63) indicated this was a personal decision and 72% (n=45) of these stated they perceived there to be benefits prior to using the Framework. Participants indicated that their use of the Framework was mainly for their own personal development plan (78%, n=94) and/or for self-assessment (65%, n=79). Participants stated that the Framework was also being used for preparation for education and training programmes (40%, n=48), assessment of the IPC team and/or service needs (39%, n=47), business case development (16%, n=19), and seminar or conference preparation (12%, n=14). The majority of participants said that they used the whole Framework (83%, n=100), with 17% (n=21) only using specific competences. Those who stated they only used specific competences said that this was dependent on the IPC activity that is occurring at a specific time. Another participant stated that as a manager, his/her staff members were asked to self-assess against four competences – two where they perceive they are performing at a high level and two where they perceive they need to improve performance. This assessment then forms the basis of staff appraisal meetings.

When asked about specific competences not used, one participant stated outbreak management, another participant stated research and another participant stated microbiology. One participant also said that only a few competences would be focused on with new members of staff to prevent them from being overwhelmed.

Analyses based on the participants’ demographic/work-related characteristics yielded some significant associations. Those working in an acute care hospital were more likely than everyone else to have received a hard copy of the Framework (52% v. 39%; χ²=4.074; p=0.044). Participants who worked in two or more healthcare organisations were more likely to have read the Framework compared to those working in only one (91% v. 74%; χ²=5.699; p=0.016). Moreover, respondents who had read the Framework and found it easy to use were those who had been working in IPC for a longer period of time (median 10 v. 6.5 years; p=0.014); (median 10 v. 5.75 years; p=0.017). This group of participants were also more likely to perceive benefits prior to using the Framework if they had been longer in a role with IP responsibilities (median 10 v. 7 years; p=0.039). Participants who were not nurses or had no managerial/directorial role were more likely to report ignorance of the Framework’s existence (32% v. 8% v. 7%; χ²=10.788; p=0.005). These were mainly medical microbiologists.

Participants with a specialist practitioner qualification were more likely to perceive benefits to using the Framework (77% v. 55%; χ²=11.600; p=0.003). Those who were working in an acute hospital were more likely than others to report that their managers were aware of the Framework (66% v. 49%; χ²=10.150; p=0.006), whereas those working in a national health protection organisation (3%) were more likely to report ignorance on their manager’s part (60% v. 14%; χ²=8.547; p=0.014). Interestingly, participants whose job title was nursing (other than manager or director) were more likely to have been encouraged by their managers to use the framework (χ²=11.324; p=0.003). Also, IPCPs were more likely to report that the decision to use the Framework was their manager’s or leader’s rather than their own compared to others (χ²=9.559; p=0.008).

Participants whose managers were aware of the Framework were more likely to be using it in practice (59% v. 19%; χ²=36.586; p<0.001), as were those who had been actively encouraged by their manager to do so (67% v. 33%; χ²=23.065; p<0.001). Managers who were aware of the Framework were more likely to have encouraged others to use it in practice (p<0.001).

In the cases where participants were unsure about the Framework’s overall benefits, it was their manager or leader who had decided on the implementation of the Framework rather than respondents themselves (39% v. 16%; χ²=12.618; p=0.046). Similarly, participants who regarded the Framework as beneficial before testing it in practice were more likely to have decided/contributed to the decision for its implementation later on (86% v. 57%; χ²=12.618; p<0.013).

Participants who were using the Framework in practice were more likely to be those who perceived no barriers in its implementation (63% v. 34%; χ²=37.053; p<0.001), and were also more likely to find benefits from its use (90% v. 39%; χ²=60.252; p<0.001). Finally, individuals who revealed that the Framework was hard to read were more likely to find no benefits or be unsure about any benefits associated with its use in practice (63% v. 21.0%; χ²=25.770; p<0.001). This is closely linked to the finding that participants who felt that the Framework was hard to read were less likely to be using it in practice (58% v. 25.0%; χ²=19.055; p<0.001), and also more likely to identify barriers preventing its use (72% v. 33%; χ²=38.452; p<0.001).

Discussion

A competency approach, focusing on ability and aptitude to create a solid foundation for workforce development is a commonly used approach in healthcare organisations (Willcocks, 2011). This Framework therefore aims to provide a solid foundation for workforce development. However, significant concerns have been raised about the extent to which such a framework improves or enhances performance both at an individual and an organisational level (Bolden and Gosling, 2006). Despite the number of competency frameworks in use, no literature could be located that examined uptake, use or impact. Consequently, this is the first and most comprehensive evaluation of a competency framework conducted to date. This study set out to identify the scope of uptake and use of the Framework, how it was being used in practice and the barriers and challenges surrounding its use.

The results show that the majority of participants were aware of the Framework’s existence, that they had accessed it either through obtaining a hard copy or electronically, had read it and found it easy to understand. This indicates that the specific communication strategies developed and implemented by the competency steering group members were successful to a certain degree. However, despite this, it is disappointing that 10% still had not heard of the Framework and 25% had not read the Framework. Reasons cited for this included competing priorities and lack of time. Such reasons were not unexpected, as roles and responsibilities of IPCPs continue to increase (Sydnor and Perl, 2011).

In relation to how the Framework was being used in practice, it was reassuring to find that some were using it to enhance structured learning and development, to identify gaps in practice and to assist with improving standards and quality care. It was also good to find that those who were using the Framework were tailoring it to their needs depending on their role and responsibility, such as choosing to use the entire competences or only focusing on specific ones. These findings also reflect the overarching aims of a competency framework in terms of ensuring individuals and teams develop the essential knowledge, abilities and skills that are needed to demonstrate effective practice, thus building a high performing organisation (Colagiuri and Ritchie, 1996; Becker, 2007; RCN, 2007; Gelling, 2008; European Centre for Disease Prevention and Control, 2008; Whittingham, 2009).

Competency development is subjective and esoteric as it relies on the assumption that human action and interaction can be broken down into its constituent parts and outcomes accurately predicted. However, while the merits of a competency framework are evident, if it is used inappropriately or not understood, it can conversely create a defensive and reductionist approach to practice (Becker, 2007). This study showed that less than half of the participants were using the Framework in practice at the time of evaluation. In addition to the competing priorities and lack of time, it was evident that, to some, there were a number of issues that prevented its use, for example that it was too long and overwhelming and that there was too much jargon included. Moreover, some individuals remained unsure of how to use it, especially in the absence of a self-assessment form.

The implementation of the Framework appears to be dependent on a number of factors, including the role of the practitioner, the number of years in service, location of work, and whether or not managers had encouraged their staff to use it. One of the most common factors that attenuates behaviour or culture change within healthcare is lack of leadership and management, whereas individuals with strong managers and leaders tend to demonstrate more positive organisational performance (Scott et al, 2003).

The overall aim of this part of the study was to evaluate the short and medium term outcomes as set out in the Outcome Logic Model in order to assist with the evaluation of the Framework’s impact and success. The findings of this study demonstrated that while the Framework is being used effectively in some areas, there are opportunities to develop strategies to further promote the use of what is clearly a valuable tool for IPCNs and those who manage them. Work is now required to explore ways in which the findings of this study can be used to further strengthen the Framework and thereby increase and enhance its use in healthcare settings. Additionally, these findings suggest that perhaps a user guide for practitioners with different roles and responsibilities could be a valuable addition.

The main limitation of this study was the relatively low response rate. The total sample size is not known due to further distribution of the questionnaire to IPC networks from wider stakeholders. However, based on the initial IPS member sample of 1,802, the response rate was 13%. This also resulted in small numbers of some of the demographic variables used for analysis. Consequently, any generalisations should be made with caution.

Conclusion

This study has found that while the Framework is being used by 43% of respondents it still needs to be used more to ensure that IPCPs are fit for purpose. Simplification, education and communication of content and purpose may well aid the goals. Although the Framework is not being used as much as intended, those who are using it are finding it useful. Further work on evaluation of the Framework will be required over time.