Multi-method evaluation of a national clinical fellowship programme to build leadership capacity for quality improvement

Abstract

Background

This paper reports the results of the evaluation of the Scottish Quality and Safety Fellowship (SQSF)—a 10-month, lead-level international educational programme established in 2008 with the overarching aim of developing clinicians with advanced quality improvement knowledge, technical ability and essential leadership skills. The evaluation explores four levels of educational and practice outcomes associated with (1) the reaction of fellows to SQSF participation, (2) learning gained, (3) subsequent behaviour changes and (4) the overall impact on national and international level capability and capacity building.

Methods

A theory-informed multi-method design was applied using (1) a search and review of the SQSF organisational database to tabulate personal, professional and demographic characteristics; (2) semi-structured telephone interviews with 16 participants using purposive and self-selected sampling; and (3) a cross-sectional online evaluation survey across all 10 cohorts involving 222 fellows

Results

SQSF was positively perceived as a high-quality learning experience containing a well-balanced mix of theory and practice, with a majority of respondents reporting career changing benefits. Most participants reported improved social, behavioural and emotional skills, knowledge and attitudes and, with sustained support of their host organisations, were able to apply and share learning in their workplace. The impact of the SQSF on a wider national and international level capability and capacity was both mediated and moderated by a wide range of interrelated contextual factors.

Conclusions

This multi-method evaluation demonstrates that SQSF has achieved significant positive outcomes for the great majority of participants. Some tentative recommendations are provided with the aim of further enhancing fellowship content, delivery, transfer and future evaluations of wider impacts at regional, national and international health system levels.

Background

In response to well-established international evidence of unwarranted variations in healthcare processes and clinical outcomes,¹ building leadership capacity and capability in quality improvement (QI) and patient safety is now a policy priority in many modern health systems, including the National Health Service in Scotland (NHSS).^2–6 Clinical leaders have the potential to influence and enact positive change and improvement within healthcare organisations, but evidence of the professional and institutional impacts of related training programmes remains.^7–9

NHSS has invested heavily in workforce QI education and training, including at the leadership level via the Scottish Quality and Safety Fellowship (SQSF) programme managed by NHS Education for Scotland (NES), in partnership with Healthcare Improvement Scotland (HIS). The fellowship is a 10-month international educational programme established in 2008 which is open to clinical professionals in Scotland and across Europe who demonstrate a strong interest in QI and leadership and have a significant role in improving the delivery of safe and effective patient care. The educational programme content consists of interactive learning sessions focusing on developing leadership skills and promoting QI principles and values, as well as residential workshops to facilitate learning from global experts, and practical opportunities to apply and spread the learning in the workplace setting and beyond (online supplemental file 1).

Objectives

Framed by Kirkpatrick’s Four-Level Model,¹⁰ the paper describes a programme evaluation study aimed at exploring four related levels of educational and practice outcomes:

• Level 1: Reaction (the degree to which participants found SQSF favourable, engaging and relevant to their jobs);

• Level 2: Learning (the degree to which participants acquired the intended knowledge, skills, confidence and commitment based on their participation in the SQSF);

• Level 3: Behaviour (the degree to which participants applied the acquired quality, safety and leadership knowledge and skills into practical implementation of QI activities); and

• Level 4: Results (the degree to which participants in the SQSF were successful in developing QI capability and capacity on wider national and international levels).

Methods

Data collection and recruitment

Review of the SQSF organisational database

A search and review of SQSF organisational database, administered by NES, was undertaken to descriptively analyse and tabulate personal, professional and demographic characteristics of current and past fellows.

Self-reported online questionnaire

Responses from routinely collected online post-programme evaluation questionnaires were collated via Questback. The survey targeted all fellows from cohorts 1 to 10 and was specifically designed for the current evaluation (online supplemental file 2). It drew on constructs assembled in previous validated instruments^{7 10 11} which were subsequently commented on clarity and appropriateness by the wider research team. This led to refinement of the survey in terms of wording, sequencing and new additional statements added. The final instrument had six sections designed to gather data about fellows’ background, their overall experiences of the fellowship, specific comments on the content, implementation and quality of the programme, views about the knowledge and skills gained, as well as insights about the potential avenues for capacity and capability building.

Semi-structured interviews

Sixteen semi-structured telephone interviews (ranging in length from 30 to 60 min) were conducted by three researchers experienced in conducting qualitative health service evaluations. The interviews aimed to provide an in-depth exploration into how fellows understood and experienced their participation in SQSF. Seven participants were recruited via purposive sampling across all cohorts (1–10) to reflect heterogeneity in terms of gender, time elapsed since graduation (7–10 years, 3–6 years, 1–2 years), country base (eg, Scotland, Norway or Ireland), care sector (eg, primary care or secondary care) and clinical profession (eg, medicine, nursing and midwifery, pharmacy). Additional interviewees (n=9) were self-selected. The topic guide was informed by two telephone interviews with programme leads and relevant published literature.^{6 8} Saturation was judged to have been reached after 16 interviews, when no new significant themes emerged. Interviews were all audiotaped with permission and transcribed verbatim to facilitate data analysis.

Data analysis

Self-completed online questionnaire

Quantitative data were subject to a basic descriptive statistical analysis using SPSS v27. The coding framework developed from the semi-structured interviews was applied to all open-ended free-text responses which were then categorised and summarised independently according to the same recurring themes. The evaluation team shared interim results with key stakeholders and encouraged participatory interpretation of the findings. Alternative interpretations were included in the analysis. Evaluation findings are presented in such a way as to capture the common themes across individual interviews and survey data sets, as well as comments that were unique to each participant.

Semi-structured interviews

Interview transcripts were analysed using the framework approach,¹² facilitated by NVivo. The first iteration of the framework was developed using a subsample of six full interviews and categories were agreed by the research team to ensure consistency in coding. After applying the framework to analyse these data, comparisons between the coded data and the Kirkpatrick Model¹⁰ and Mery’s framework⁷ were made using a theory-based content analysis approach.¹³ Three interviews were double-coded to ensure consistency and the researchers debriefed the wider team regularly.

Patient and public involvement statement

Downstream outcome assessment was never the intention of this evaluation. Although the methods employed with the current sample were as rigorous as possible given available resource, the design of the evaluation meant that we did not speak to health service users on whom educational or improvement interventions may have impacted. Therefore, the findings did not include consideration of their priorities, experiences and perspectives in terms of the impact that the programme has had on the quality of care received.

Findings

Participants’ characteristics

Self-reported online questionnaire

Of the 222 participants in cohorts 1 to 10 (table 1), the majority were Scottish based (n=143, 64.0%), female (n=129, 58.1%) and medical doctors (n=138, 62.0%). The nursing, midwifery and allied health professions were the next largest grouping (n=59, 26.6%), while participants from Northern Ireland (n=34, 15.0%) constituted the second largest national grouping. The great majority of participants were based in the acute setting (n=191, 86.0%).

Table 1.

Personal and professional characteristics of Scottish Quality and Safety fellows

Factor	N (%) Cohorts 1–10 (total n=222)
Gender
Female	129 (58.1)
Male	93 (41.2)
Professional grouping
Medical doctors (excluding GPs)	138 (62.0)
General medical practitioners	6 (2.7)
Nursing, midwifery and allied health professionals	59 (26.6)
Pharmacists	9 (4.0)
Dentists	1 (0.5)
Others	3 (1.4)
Unknown	6 (2.7)
Care sector base
Acute hospital setting	191 (86.0)
Primary care	12 (5.0)
Community	12 (5.0)
Mental health	0 (0.0)
Other	7 (3.0)
Country
Scotland	143 (64.0)
Northern Ireland	34 (15.0)
Norway	15 (7.0)
Ireland	14 (6.0)
Denmark	13 (6.0)
Wales	1 (0.5)
England	1 (0.5)
Canada	1 (0.5)
Fellowship cohort
1	6 (3.0)
2	12 (5.0)
3	13 (6.0)
4	16 (7.0)
5	22 (10.0)
6	29 (13.0)
7	29 (13.0)
8	30 (14.0)
9	32 (14.0)
10	33 (15.0)

A total of 93 completed questionnaires were returned for analysis across 10 cohorts, equating to a response rate of 44.2% (93/210) with 12 emails deemed as undeliverable. Of the 93 respondents, the majority were Scottish based (n=59, 63.4%) and medical doctors (n=85, 91.3%), working across 7 of the 14 NHSS territorial health boards (table 2).

Table 2.

Crude comparison of the proportion of Scotland-based fellows recruited from NHS territorial boards

NHS Board	NHS Board employees (n)	SQSF fellows n (%)
Ayrshire and Arran	5783.2	13 (0.22)
Borders	1615.2	8 (0.5)
Greater Glasgow and Clyde	22 015.9	26 (0.11)
Fife	4811.2	6 (0.12)
Lothian	13 813.5	22 (0.16)
Highland	4336.7	2 (0.05)
Grampian	7658.5	13 (0.17)
Orkney	286.2	1 (0.35)
Tayside	7215.3	11 (0.15)
Western Isles	506.6	1 (0.2)
Shetland	311.5	0 (0.0)
Lanarkshire	7421	12 (0.16)
Forth Valley	3784.4	9 (0.24)
Dumfries and Galloway	2277.4	3 (0.13)
All	81 836.6	*143 (0.17)

*Additional 16 fellows are based in special health boards or at Scottish Government.

Semi-structured interviews

Sixteen participants were interviewed for the purpose of this evaluation. The sample contained equal numbers of females (n=8) and males (n=8); 12 participants were Scottish based, three from Northern Ireland and one from elsewhere: eight were hospital consultants, five were nurses, and one GP, pharmacist and allied health professional were represented.

Level 1: Rreaction

The SQSF programme was positively perceived as a high-quality learning experience containing a well-balanced mix of theory and practice, with a majority of fellows reporting clinical practice and career changing benefits. Between 76% and 90% of survey respondents agreed or strongly agreed with statements indicating their satisfaction with programme content, delivery and format or indicating the extent to which they valued key course components (table 3).

Table 3.

SQSF fellows’ views on the quality of the fellowship programme (levels of agreement ≥4 on the rating scale)

Attitudinal statements	Scotland (=59)	Non-Scotland (n=34)	Total (n=93)	Kirkpatrick level
	n, %	n, %	n, %
I was satisfied with the reading materials and resources provided	53 (89.8)	33 (97)	86 (92.4)	1
I was satisfied with the presentations by Subject Matter Experts	53 (89.8)	34 (100)	87 (93.5)	1
I was satisfied with how the programme content is relevant to my work role	54 (91.5)	34 (100)	88 (94.6)	1
I was satisfied with the format of the overall programme (eg, the balance between presentations, group work activity and self-learning etc)	52 (88.1)	34 (100)	86 (92.4)	1
Overall, I thought the fellowship programme was high quality in terms of content and organisation	56 (94.9)	34 (100)	90 (96.7)	1
Promoting Stakeholder engagement (how to strengthen existing collaborations within NHS Scotland/Your country and beyond and actively engage key stakeholders at all stages of the improvement process)	45 (76.2)	29 (85.2)	74 (79.5)	2
Embedding sustainability, scale and spread (how to plan for and assess the sustainability, scaling up and spreading a long-term quality improvement and patient safety culture)	44 (89.7)	30 (88.2)	74 (79.5)	2
Supporting Evaluation (how to monitor and evaluate implementation of improvement interventions and assess outcomes)	46 (77.9)	32 (94.1)	78 (83.8)	2
The Fellowship Programme has enhanced my knowledge about leadership and quality improvement more broadly	56 (94.9)	33 (97)	89 (95.6)	2
The Fellowship Programme enhanced my self-confidence to practice quality improvement and improve patient safety	57 (96.6)	31 (91.1)	88 (94.6)	2
Since participating in the Fellowship Programme, my practice has changed in terms of undertaking and leading quality improvement activity	53 (89.8)	32 (94.1)	85 (91.3)	3

Interviewees concurred that the course objectives were clearly communicated, and the content and delivery methods were locally, professionally and developmentally relevant. External contributors and project leads were perceived as high calibre. Practical application of acquired knowledge within interactive team-based learning during the residential components was mostly valued.

The Fellowship Programme was the best run and enjoyable course I have ever been on. It was a life changing experience that completely changed the way that I practice medicine and made me a better doctor. Just the right balance between learning, challenging us and having lots of fun. Speakers were inspirational and the residentials hugely helpful in taking time out of work to fully immerse yourself in all that the fellowship offers. (I15, Hospital Consultant)

Level 2: Llearning

SQSF was successful at consistently increasing awareness, knowledge and skills for most participants. Between 89%–96% of survey respondents indicated very strong agreement with statements regarding impacts on capability and confidence to practice (table 3).

Interviewees also reported the SQSF deepened their understanding of important concepts, theory and methods applied as part of QI and patient safety efforts:

My skills around using QI methodology have massively improved. I’ve moved from being quite narrowly focused on my specific clinical area to having much broader horizons, and having the knowledge, skills and confidence to work on bigger service developments in areas that have been notoriously unimproved and unimprovable. (I14, Nurse)

While interviewees valued the domain knowledge/skills relating to practical QI approaches, many reported the content and practice opportunities relating to social, behavioural and emotional skill development as being the most impactful aspect. This has helped them become more confident clinicians who are now willing to take appropriate risks, make mistakes and learn from failures:

I’m much more philosophical when there’s problems at work so I’m trying to give people time and space to reflect on the problems that they’ve encountered. It’s something developed in the Fellowship that’s gone into my clinical arena and made me a better clinician overall. (I7, Hospital Consultant)

Several participants described an increased ability to engage in difficult conversations in a comfortable and structured way as well as an improved ability to voice concerns constructively. Using words such as “energized,” “inspired” and “invigorated, they described how residential and networking opportunities allowed access to world-class experts, enhanced their self-efficacy and confidence, and increased their ability to influence senior gate keepers.

The opportunity to network and make connections, the comrade support from an ever-expanding family of fellows and experts assisted with personal growth in terms of breaking down professional boundaries and removing stereotypes so I would say, stick your head above the parapet and forget about the impostor syndrome. (I4, Allied Healthcare Professional)

Other fellows described important attitudinal shifts related to increased appreciation of diversity and openness to change, and a desire to challenge and push themselves outside their comfort zones and be a catalyst for change:

It is about a subtler attitudinal change—a core belief in the power of individuals to transform—both patients and staff. I’ve come to believe that if people are shown some tools and given some space and self-belief they will become change agents. (I2, Nurse)

Level 3: Behaviour

Most participants had improved their knowledge and, with sustained support and encouragement of their host organisations, were able to apply the knowledge and skills learnt to their workplace. Seventy-six per cent of survey respondents indicated agreement or strong agreement with statements about spreading their learning locally either informally with colleagues or more formally via training, teaching or coaching. Furthermore, 86% of respondents agreed or strongly agreed that their SQSF project had achieved its goals and 79% that their projects and related improvement work were still ongoing in their workplace (table 4).

Table 4.

SQSF fellows’ views on building improvement leadership capacity and capability (levels of agreement ≥4 on the rating scale)

Attitudinal statements	Scotland (=59)	Non-Scotland (n=34)	Total (n=93)	Kirkpatrick level
	n, %	n, %	n, %
Sharing Fellowship Programme course materials (eg, slides, activity sheets, resources) with colleagues within your organisation	39 (66.1)	26 (76.4)	65 (69.8)	3
Formal training of groups of people in your organisation on how to apply theories, models and tools learned during the Fellowship Programme	38 (64.4)	22 (64.7)	60 (64.5)	3
Coaching and/or mentoring of individuals in your organisation on how to apply theories, models and tools learned during the Fellowship Programme	45 (76.2)	26 (76.4)	71 (76.3)	3
Having taken part in the Fellowship programme, I have been involved in training and support others (ie, teaching) and therefore the spread of QI theory and practice:
Locally	47 (79.6)	28 (82.3)	75 (80.6)	3
Nationally	34 (57.6)	17 (50)	51 (54.8)	4
Internationally	18 (30.5)	3 (8.8)	21 (22.5)	4

Semi-structured interview data suggested most participants learnt about and implemented specific QI strategies, while others redefined or tailored their roles and aims as clinicians. Tacit knowledge, new ideas, innovations and motivation resulting from the fellowship were all often transferred into practice:

I learnt that I had to find the ‘heart’ in my work and understand what part of my work really enthused me. My career then changed direction. I have trained as a coach, facilitated regional workshops and focused on development as a senior educator—because this is how I am best suited to make a difference. (I11, Hospital Consultant)

Between 80% and 82% of survey participants claimed to have had some form of influence on local capability and capacity building (table 4). Four common components of this knowledge transfer process were illustrated through interview data. First, teaching QI and related curriculum development were found to be the most frequent influence.

I am scheduled to teach my consultant colleagues this autumn. I have already taught at university level, I have led QI teaching sessions to local trainees and colleagues and have shaped the content of the national registrar training programme. (I12, Hospital Consultant)

Second, many of the fellows mentioned a shift in identity in relation to mentoring colleagues in developing their own QI capability.

I am now doing a lot of formal and ad hoc coaching and mentorship of others, especially around clinical audit and key performance indicators, data for Improvement and understanding variation. (I14, Nurse)

Third, some participants indicated that the fellowship helped them address their specific professional development needs and actively seek out further educational opportunities:

SQSF provided a watershed moment for pursuing further education. It was like letting the genie out of the bottle. It introduced me to a broad and varied literature and key authors who have shaped my thinking and made me explore my interests further within a PhD. (I10, Pharmacist)

Finally, fellows reported being involved in a wide range of QI-related spread and dissemination activities at the local, national and international levels (table 5). This included contributing to the science of improvement through publishing articles in peer-reviewed healthcare journals, presenting at national and international conferences, winning innovation awards and securing funding for future improvement work.

Table 5.

SQSF fellows’ reported professional achievements and impacts (n=93)

Achievement and impact	n	Kirkpatrick level
Publishing improvement work in a peer-reviewed healthcare journal	25	4
Presenting a poster at a regional, national or international conference	57	3/4
Delivering a talk at a regional, national or international conference	61	3/4
Winning an award for an improvement project	24	4
Winning an award for leadership in safety and improvement	4	4

Level 4: Rresults

Several participants described how taking part in SQSF reinvigorated their career through attaining a more senior position with a designated QI national or organisational role, which often involved a change from ‘doing’ to ‘leading’ others:

I am now able to influence more system wide change and will continue to inspire a network of improvers and innovators who have knowledge, skills and contacts to lead change and drive QI agenda in Scotland and beyond. (I07, Hospital Consultant)

Other fellows described how they contributed to establishing QI infrastructures such as QI academies that drive further related capability and capacity opportunities.

I support most of delivery of the formal QI training locally and nationally. We’ve identified that there wasn’t much capacity in the organisation, so we’ve been sending people on a variety of courses especially through our links with NES platforms which provide staff with access to learning and practice support resources. (I06, Nurse)

A minority of fellows have established or scaled national or international projects. Some examples include becoming active partners in strategic international collaborations with the aim of improving health and social care, at a significant scale and pace:

I became a National Clinical Lead shortly after I finished my Fellowship. I am now advising a wide range of international partners in development of QI strategies, including Brazil, Canada and Australia. Allegedly I am now viewed as a QI expert and I feel like I am the ‘go to’ person for QI advice. (I16, Hospital Consultant)

Organisational level contexts and support influenced fellows’ ability to increase QI capability and capacity in their organisation or beyond. For instance, a prominent theme during the interviews was work pressures, time constraints and backlog of work resulting from attending training:

How to prioritise your priorities when everything is a priority? It can look daunting and overwhelming. I’ve got my toe in the water but obviously that means stepping away from the involvement in your day-to-day clinical role. (I08, General Practitioner)

Another concern related to a lack of understanding of fellows’ potential contribution among senior leaders, failing to involve them in tackling organisational challenges and ultimately leading to their under-utilisation.

I felt like we got to the end of the programme and we were on our own. It was made harder by returning to a place who didn’t seem to know quite what to do with me! At the end of the Fellowship I felt empowered, skilled and motivated and within a few weeks I felt isolated, frustrated and discouraged. (I09, Nurse)

It was clear from participants’ interviews that the fellowship programme was unlikely to have a lasting effect without sustained organisational support. A supportive ethos and an expectation that all healthcare professionals proactively engage in QI have been found to be important factors in securing change as a result of SQSF:

How to change the leadership culture and the dominant command and control hierarchy focused strongly on performance management? Leading within one’s own team is hard unless one has the executive lead firmly on one’s side, because others within the team want to jockey for their own positions. It is harder to be a prophet in your own land. (I05, Hospital Consultant)

Discussion

Summary of main findings

The evaluation achieved its aims of exploring four levels of educational and practice outcomes associated with SQSF participation, learning gained, subsequent behaviour changes and overall impacts on national and international level capability and capacity building. Findings from the semi-structured interviews and online survey were complementary and generated confirmatory results, showing that, overall, SQSF achieved positive outcomes at the Reaction, Learning and Behaviour evaluation levels. There was self-reported evidence of the numerous and diverse personal and professional benefits of SQSF participation at level one (Reaction), which was described as a high-quality learning experience with valuable, well-tailored content and delivery methods. Regarding Learning at level 2, a large majority of participants believed that the SQSF programme has provided them with the social, behavioural and emotional skills and knowledge to advance and lead the organisational QI agenda and given some a new-found confidence and ability to engage, collaborate and influence others. Furthermore, most participants were reportedly able to apply the knowledge and skills learnt in the SQSF to their workplace and wider organisations (level 3 Behaviour Change), particularly through undertaking a QI project, teaching quality improvement and curriculum development, mentoring colleagues in developing their own capability, pursuing further education and becoming actively involved in a wide range of dissemination activities. It was however more challenging to establish strong or consistent solely SQSF attributable evidence of level 4 (Organisational Performance) on wider national or international outcomes. Overall, the findings offer a small knowledge contribution of evaluation of QI training at the healthcare ‘system-level’ (where evidence is very limited) in contrast to most similar assessments which are “…restricted to specific training programmes on single sites or a clinical condition or process that represents one particular organisation problem”.⁷

Strengths and limitations

Key strengths included a strong historical commitment from the SQSF leadership and support team to routinely collect and collate good quality programme and evaluation data, which was made available for this study. The complex and shifting nature of fellows’ experiences meant that they benefit from their participation in SQSF in diverse ways, offering programme leadership and other stakeholders’ potential reassurance that this system-level educational intervention is well received. For instance, our findings largely support the overall approach adopted by SQSF in providing dedicated teaching faculty and using multiple and diverse QI teaching strategies that improve related educational outcomes such as learner experiences, empowerment and QI confidence levels.^6–9

Important evaluation limitations included the largely descriptive nature of the study, relying mainly on self-report data. This means the data analysed are likely to be affected by responder bias while reports of learning and improvement cannot be verified independently. Another imitation is that those interviewed did not represent the overall study sample, but purposive sampling attempted to reflect key characteristics of interest. Furthermore, there is a need to broaden the sample representation beyond clinicians based in acute hospital settings. For instance, although 90% of interactions with the NHS start and finish in primary care¹⁴ this sector may be under-represented on the programme and is potentially missing out on building much-needed QI leadership capacity and capability. Lastly, the sample also reflects a possible imbalance in the representativeness of Scotland-based participants, and the proportions of participants applying and being selected from across territorial health boards with particularly well-developed QI infrastructures compared with others. However, developing capacity and capability for improvement encounters some similar challenges regardless of the context, so we believe that lessons described in this evaluation will be more widely pertinent, including in organisations delivering similar programmes beyond healthcare.

Recommendations for educational development

First, given that successful acceptance onto SQSF is through a competitive process, the challenge for stakeholders is to retain this competitive element while promoting, encouraging and ‘incentivising’ the benefits and opportunities associated with the programme to those professional groups who historically have low engagement with this type of education for a variety of practical, cultural and contractual reasons (eg, independent clinical contractors in primary care).

Second, as a minimum, efforts should be made to develop and implement a systematic front-end analysis process to identify educational and training needs and diagnose organisational transfer barriers before the fellowship is designed and delivered.^15–20 Similarly, there might be a need to communicate individual and organisational outcome and impact expectations to all stakeholders before SQSF begins.^15–18 Consolidating post-fellowship mechanisms can also be useful so that participants can continue to network and collaborate with peers and colleagues about successes or challenges in creating and sustaining improvements.⁷

Third, our evidence suggests that despite overwhelmingly positive reactions and learning, similar to other comparable QI educational programmes,^{7 11} the extent to which participants were able to transfer training into the workplace was mediated by a wide range of interrelated situational factors such as internal motivations, supportive organisational culture, available resources and further coaching and feedback on an ongoing basis.^15–17 Respondents’ descriptions of their support needs are informative and provocative and can assist organisational leaders and managers in supporting and facilitating their professional learning. Managers should be prepared to actively support fellows and be equipped with a sound and fundamental understanding of how fellows learn to cultivate their interests, how they navigate the interactions among the various components of the SQSF, and how their professional activities interact, relate and shape learning.^{15 16} Sending clinicians on a fellowship programme just because they are available is not likely to be effective in the longer term.^{7 9 11} Fellowship provision is arguably more effective if attention is given to ensuring that the work climate and management practices encourage personal development, while ongoing coaching and feedback facilitates the use of goal-setting and action planning for learning, knowledge transfer and impact.7 11 15–20

Finally, Dixon-Woods and Martin outline key challenges for the policy, organisational and research communities in improving the quality of QI efforts in healthcare.²¹ However, these important issues are equally valid for healthcare and academic educators and arguably should be included in the content design of QI training programmes to raise awareness of, and address, the following complex issues during the training process. For example:

• QI as an improvement activity is hugely problematic—the evidence it improves care processes and outcomes evidence is mixed, even when including the more rigorously conducted studies.

• The fidelity in application of QI methods by healthcare professionals, such as PDSA cycles (an example of an ‘industrial’ QI approach²²) or criterion-based audit (an example of a ‘professional’ QI approach²³) is also variable and may be a significant reason why the evidence base is so mixed.

• Evaluations of QI as a method to share learning are of low quality and need to be a higher priority and conducted more rigorously.

• It is assumed (erroneously) that QI approaches ‘will work’ in any situation regardless of the clinical and socio-cultural contexts encountered—the variable implementation and impact of the WHO safety checklist is a significant case in point.²⁴

• Much improvement activity is localised and does not tap into broader organisational resources including opportunities to tackle common problems more collectively and so design more meaningful solutions.

Recommendations for research and evaluation

The Kirkpatrick model provided an appealing theoretical framework for organising the evaluation process and furnished important means for assessing factors essential to fellowship effectiveness. It offered a systematic approach to training evaluation that did not necessitate baseline measures and provided a focus on transfer of learning beyond the shorter-term and individual benefits.²⁵ The results were a more complete and accurate understanding of the ‘why and wherefore’ of evaluative outcomes, and a firm foundation for making useful recommendations for both researchers and evaluators, who might further interpret the findings considering their experiences, contexts and practice.

Despite its advantages, the use of Kirkpatrick model was not without limitations. Although level four outcomes are generally seen as of utmost importance, evaluators and theorists, including Kirkpatrick, generally acknowledge that this type of assessment is not without challenges and tends to be avoided.^{10 26 27} Similar to the earlier evaluation of this programme over 5 years ago¹¹ and reflecting the current debates in the literature,^{28 29} we have not been able to robustly evidence that all levels are causally related, interdependent and accumulative or that they ultimately lead to organisational and wider professional or system impacts. Furthermore, even if level four data is to be achieved, it will be over simplistic to attribute behaviour changes and resulting system level impacts solely or directly to ‘participating in SQSF alone’. This poses important questions about whether the scarcity of level four evaluations is due to limited resources, lack of robust contextual information or difficulties in harnessing, exploiting and developing effective data linkage.^{25 30 31} For instance, having access to robust clinical outcome data from project improvement work may have provided more penetrating insights and enabled an element of verification of success or otherwise. Better data linkage between different quantitative and qualitative sources and case studies may have also aided understanding of the facilitating and hindering individual, organisational and wider health system contexts that should be considered as an adjunct to training delivery and evaluation.³² Indeed, while the wider system impact of taking part in the SQSF was less well evidenced, this was nonetheless found to be conditional on the opportunities and resources available to use new knowledge and skills effectively. For any large and relatively expensive programmes such as SQSF, the investment of time, money and participant energy is necessary to enable participants to fully flourish in their host organisations; however, a further limitation of this and most evaluations of QI capability building is the lack of robust data on the ‘return on investment’.⁷

Conclusion

There is national and international recognition of the ever-greater need for clinical leadership capacity and capability to implement practical change and enhance the effectiveness of patient safety and quality improvement activity in NHSS and beyond. The SQSF was envisioned, specifically designed and delivered with the intention to fulfil this need. The findings provide affirmative evidence of the educational utility of the SQSF in terms of its acceptability to participants, its practical feasibility as a long-standing national training programme and the multiple self-reported personal, professional and organisation benefits associated with participation. The data also suggest that participants who are medically qualified, male and based in the acute hospital setting are potentially over-represented when crudely compared with related national workforce statistics, which is a possible issue for exploration by relevant stakeholders, along with time-management and organisational support challenges to achieving wider impact for some. These potential areas for improvement should be of particular interest to those leading on designing, delivering and evaluating QI education at organisational, regional and national health system levels in the UK and worldwide.

List of abbreviations

NHS Scotland

NHSS is the public funded healthcare system in Scotland. Approximately 160 000 staff work across 14 regional NHS Boards, seven Special NHS Boards and one public health body.

Healthcare Improvement Scotland

HIS is a Special NHS Board which supports the healthcare priorities of the Scottish Government, in particular the Healthcare Quality Strategy and the 2020 Vision and the Scottish Patient Safety Programme.

NHS Education for Scotland

NES is the national Special Health Board with a responsibility to develop and deliver education and training for those people who work in NHS Scotland

Safety, Skills and Improvement Research Collaborative

SKIRC is a cross-directorate, multi-disciplinary body with a focus on research, innovation and evaluation in key areas of strategic importance to Scottish Government, NES, territorial NHS boards, higher educational institutes, professional bodies and regulators—Patient Safety, Clinical Skills, Simulation, Quality Improvement and Human Factors/Ergonomics.

Scottish Improvement Science Collaborating Centre

SISCC is funded to strengthen the evidence base for improving the quality of care sustainably and at scale. The SISCC programme of work aims to generate new knowledge to support improvement activities within health and social care; building capacity and capability through a cross-sectoral platform. Our five research themes are as follows: Improvement Methods; Context; Spread and Sustainability; Capacity & Capability; Behaviour Change.