Success and limiting factors in health service innovation: a theory-generating mixed methods evaluation of UK projects

Abstract

Objectives

To explore and explain success and limiting factors in UK health service innovation.

Design

Mixed methods evaluation of a series of health service innovations involving a survey and interviews, with theory-generating analysis.

Setting

The research explored innovations supported by one of the UK’s Academic Health Science Networks which provides small grants, awards and structural support to health service innovators including clinical academics, health and social care professionals and third-sector organisations.

Participants

All recipients of funding or support 2014–2018 were invited to participate. We analysed survey responses relating to 56 innovation projects.

Results

Responses were used to conceptualise success along two axes: value creation for the intended beneficiaries and expansion beyond its original pilot. An analysis of variance between categories of success indicated that participation, motivation and evaluation were critical to value generation; organisational, educational and administrative support were critical to expansion; and leadership and collaborative expertise were critical to both value creation and expansion. Additional limiting factors derived from qualitative responses included difficulties navigating the boundaries and intersections between organisations, professions, sectors and cultures; a lack of support for innovation beyond the start-up phase; a lack of protected time; and staff burn-out and turnover.

Conclusions

A nested hierarchy of innovation needs has been derived via an analysis of these factors, providing targeted suggestions to enhance the success of future innovations.

Strengths and limitations of this study.

• A strength of this research is that it compares many innovations in a consistent way, and that it provides insights across a range of categories of success.

• A limitation of this research is that it is situated in a single geographical context; however, repeating these methods in different contexts should produce locally relevant results.

• Few mid-level theories relating to innovation are grounded in data that include projects that have not achieved their intended outcomes; therefore, we may have identified novel insights.

• Many of the success factors we have identified are not unique to this study; however, they have been subjected to further statistical analysis and found to differentiate significantly across categories of success in this context.

• More research is needed to examine whether addressing these factors prospectively enhances the success of future innovations.

Background

Healthcare systems worldwide are faced with increasing demand linked to the rising burden of disease within a resource-constrained environment.¹ This has led to a pressing need to find and disseminate innovative ways of meeting the healthcare needs of patients and communities in ways that are more sustainable.² The WHO characterises health service innovation as ‘a novel set of behaviours, routines, and ways of working that are discontinuous with previous practice, are directed at improving health outcomes, administrative efficiency, cost-effectiveness, or users’ experience and that are implemented by planned and coordinated actions’³ (p 7).

Academic efforts in the health sciences continue to sharpen the focus on impact, rather than the creation of ‘new knowledge’ as the primary goal of research activity. At the vanguard are implementation scientists who work to translate research and innovation into clinical practice, navigating institutional, organisational, structural and cultural complexities to improve services.⁴ New support structures have emerged, such as the 15 Academic Health Science Networks set up in 2013 by the National Health Service (NHS) England, with funding streams that aim to support and encourage innovation at various levels.⁵ After more than half a decade of programme grants, the impact of these innovation programmes is a legitimate subject of enquiry: how and why have certain innovations become normalised, sustained or spread, and why have others struggled or stopped?

The knowledge created through an individual innovation is likely to be complex and context dependent, providing insights that may not necessarily be generalisable.⁶ Meta-analyses are faced with the complexity of synthesising multiple project evaluations that may be reported in different ways. The published literature on health service innovation contains few analyses of unsuccessful innovations, despite attempts to encourage negative reporting.⁷ By evaluating a large corpus of projects across one of these academic health sciences networks in a systematic way, we have an opportunity to directly compare innovations including those that may have struggled or stopped and not reached the literature.

This study thus sets out to explore a large number of innovations, both as individual projects in their unique local contexts, and as part of a larger integrative study. By isolating the factors that differentiate between categories of success, our aim is to produce an empirically derived explanatory model, and thereby to inform and enhance the success of future innovations.

Methods

Research aim

To explore and explain success and limiting factors in health service innovation.

Methodological orientation and theory

This study is situated at the intersection of policy, social sciences and organisational research. Our philosophical assumptions are that there are real differences in the success of innovations, but also that success is fundamentally a subjective construct. Any research will only produce an approximation of the truth, and findings must be interpreted with an appreciation for context. We therefore position this research at the boundary of critical realism and constructivism.⁸

Adopting Varpio et al’s terminology on the philosophy of research, we are taking an inductive approach that works towards a theoretical framework, rather than applying a pre-existing theoretical framework to this study.⁹

We have adopted what Creswell et al refer to as a sequential mixed methods design.¹⁰ According to Creswell, insight can emerge from exploring first through qualitative methods (in our case a published qualitative review and interviews) the types of factors that might be important, and then designing questionnaires to explore their salience to a population (called an ‘exploratory sequential design’). Insight can also emerge from collecting survey data initially and then following up with interviews to help explain the survey results in more detail: an ‘explanatory sequential design’. Where both qualitative and quantitative data are collected simultaneously, one set of data can be used to triangulate the other (eg, where the meaning of one is unclear), or they can be used in complementary ways to illuminate each other (eg, one determining which factors are important, the other illuminating why that might be). Our research process involves both exploratory and explanatory aspects as well as triangulation and illumination. It is summarised in figure 1.

Figure 1.

Research overview. A mixed methods sequential research process to explore and explain success factors and limiting factors in health service innovation.

Context

The Health Innovation Network (HIN) is one of the nationally funded Academic Health Science Networks set up by NHS England in 2013. It provides small grants, awards and structural support to academics, health and social care professionals and third-sector organisations, supporting service-level innovations to improve outcomes and value, including the sustainable use of resources. In addition, in the years 2014–2017, Health Education England (South London) provided investment in training and education innovation projects across healthcare settings in South London, through its strategic investment programme.

Participants and sampling

All recipients of HIN funding and Health Education England (South London) strategic investment programme funding and support during the years 2014–2018 were invited to participate. As our sample size was moderate, we aimed to analyse all responses and retrospectively determine whether the sample size was sufficient for thematic saturation and statistical inference. We achieved a priori thematic saturation for success factors (exemplar comments for each significant factor that we found) and inductive thematic saturation for limiting factors (content coded until no new themes arose).¹¹

Research team and ethics

The research was commissioned by HIN in collaboration with Health Education England and conducted by an independent research team at King’s College London. The research team comprised a postdoctoral educational psychologist/learning scientist (GBR), a postdoctoral occupational psychologist/health services researcher (AK) and a medical education research fellow (KLG). None were in a position of power or influence over participants, and the research was carefully designed to be conducted at arm’s length from the funding agency. Survey responses were collected anonymously and decontextualised by the research team to encourage innovators to comment critically and safely about their projects. Innovation funding was not conditional on taking part in this research. Ethical approval was granted on 26 March 2019 by the Research Ethics Committee of King’s College London (LRS-18/19–10432). Written informed consent was obtained from interviewees. Consent was implied from participation in the survey.

Patient and public involvement

No patient was involved. The primary stakeholders in this research were health service innovators who were involved in the survey design and in checking back and refining our interpretation.

Data generation methods

Survey design

The survey design began with the extraction of potential success factors for health service innovation from a recently published qualitative systematic review.¹² This review aimed to identify all the factors and theories associated with sustainability and scale-up (capacity building) of innovations in health services research. KLG validated and expanded these factors through scoping interviews with five experienced health service innovators. The interviews started with an open exploration of what the innovator felt had impacted on the success of their project, followed by discussion on the factors identified through the literature. Personal factors were mentioned by all stakeholders in addition to the factors from the review, suggesting these may be under-reported. An additional theme (personal factors) with related subfactors was therefore included, based on these interviews. Themes and factors are listed in figure 2. These were used to create a mixed methods nested design survey¹³ using Qualtrics software (full text in online supplemental data).

Figure 2.

Survey scope. The survey scope was based on a qualitative review of theories and findings relating to the sustainability and scale-up of health service innovations¹² supplemented by five scoping interviews. The questions are listed in table 1, and the full-text survey is in the online supplemental material.

The survey asked respondents to:

1. Categorise and describe their project’s current status (no longer running/likely to finish soon/stable at the level of the original pilot/scaled up beyond the original pilot/too early to say/other).

2. Score statements (listed in table 1) relating to the impacts of each factor on their project’s outcomes, grouped into nine themes on a 5-point disagree/agree Likert scale.

3. Describe the status of their project and provide qualitative insights into each of the nine themes.

Table 1.

Analysis of variance of potential success factors across categories of success

	Factor	Average answer across all categories*	Distribution of factor is the same across categories of expansion (proven value)	Distribution of factor is the same across categories of expansion (all)	Distribution of factor is the same across categories of realised value	Interpretation and comment on secondary analysis‡
			Significance†	Significance†	Significance†
1	The initiative was designed to end once a set outcome had been achieved.	↘ −0.40	0.274	0.172	0.317
2	The initiative was designed to end after defined period of time.	↘ −0.58	0.011	0.011	0.288	Unsurprisingly, even projects with high-realised value finish if they are time bound.
3	The initiative was designed to address an important healthcare need.	↑ 1.63	0.765	0.712	0.135
4	There was public/political recognition and concern for the problem that the initiative was designed to address.	↑ 1.23	0.939	0.66	0.201
5	The initiative was based on a strong evidence base, and it was credible that the stated benefits could be achieved through the project plan.	↑ 1.35	0.696	0.355	0.299
6	The project was sufficiently funded.	↗ 1.17	0.941	0.454	0.126
7	The project had sufficient infrastructure, such as buildings, office space, materials or supplies.	↑ 1.52	0.952	0.842	0.613
8	There were sufficient members of staff with the right skills to meet the requirements of the initiative.	↑ 1.42	0.003	0.002	0.013	Skilled workforce is a critical success factor across all definitions of success.
9	Members of staff had sufficient energy and time to dedicate to the initiative.	↑ 1.21	0.033	0.085	0.362	Time and energy are critical to whether proven innovations expand.
10	There was sufficient administrative support to deliver and maintain the initiative.	↗ 0.85	0.013	0.019	0.142	Administrative support is critical to whether an innovation expands.
11	There was sufficient technical support to deliver and maintain the initiative.	↗ 1.04	0.113	0.187	0.657
12	There was sufficient educational support to deliver and maintain the initiative.	↑ 1.30	0.023	0.012	0.089	Educational support is critical to whether an innovation expands.
13	External political or societal factors impacted negatively on the delivery of the initiative.	↘ −0.91	0.191	0.141	0.005	External political or societal factors appear critical to whether an innovation is able to realise its intended value (inconsistent exposure/response).
14	It was necessary to adapt the project so that it aligned more closely with external political or societal priorities.	↘ −0.72	0.541	0.252	0.064
15	We had opportunities to demonstrate the benefits of this innovation within our organisation and/or to other organisations.	↑ 1.59	0.237	0.053	0.02	Unsurprisingly, innovations that were able to realise their intended value were more likely to be able to demonstrate the benefits of their innovation.
16	Steps were taken to raise the profile of the initiative, for example, through media, marketing, community engagement or publications.	↗ 0.85	0.108	0.059	0.306
17	There are plans to replicate this innovation at other sites or spread it to other parts of the organisation.	↗ 0.58	0.024	0.012	0.228	Unsurprisingly, innovations that have become scaled up were more likely to say there were plans to spread their innovation.
18	The initiative integrated well into existing organisational structures, programmes or policies.	↑ 1.26	0.012	0.002	0.059	The ability of an innovation to integrate into existing organisational structures may be critical to whether it becomes scaled up.
19	It was necessary to adapt the initiative so that it achieved a good fit with existing organisational structures, programmes or policies.	→ −0.09	0.053	0.035	0.115	For innovations to scale up, they may need to adapt so that they fit within existing organisational structures.
20	The host organisation was ready and able to undertake the initiative.	↑ 1.55	0.262	0.168	0.721
21	The initiative was hampered by opposition from within the host organisation.	↓ −1.50	0.037	0.027	0.398	However valuable an innovation is, it appears unlikely to survive if it is opposed within the host organisation.
22	The host organisation lacked the necessary values/culture to support and sustain the initiative.	↓ −1.17	0.265	0.247	0.888
23	I was released from other duties so that I could implement this initiative.	↘ −0.43	0.732	0.893	0.789
24	I had a supportive peer network that I could discuss any issues or problems with.	↑ 1.32	0.385	0.562	0.79
25	I was internally motivated to implement this initiative.	↑ 1.81	0.425	0.129	0.034	Innovations appear more likely to realise their value if the innovator is internally motivated.
26	I found working on the initiative personally rewarding.	↑ 1.81	0.147	0.067	0.022	Unsurprisingly, there is a correlation between an innovation realising its value, and the innovator finding it rewarding.
27	I feel I had the right skills/experience/training to implement and sustain the initiative.	↑ 1.62	0.033	0.008	0.023	The skills of the innovator appear to be a critical success factor across all definitions of success.
28	I had sufficient energy and time to dedicate to the initiative.	↗ 1.06	0.209	0.268	0.498
29	The project had sufficient input from experts with the necessary knowledge and experience.	↑ 1.66	0.134	0.018	0.021	Expert input appears critical to both realisation of value, and to whether it expands.
30	The outcomes and impact of the project were measured or assessed.	↑ 1.37	0.060	0.125	0.119
31	We were able to demonstrate the effectiveness of the project.	↑ 1.38	0.463	0.185	0.015	Unsurprisingly, innovations that were able to realise their intended value were more likely to be able to demonstrate the effectiveness of their innovation.
32	Performance data were gathered and reported on a regular basis.	↗ 1.04	0.346	0.266	0.159
33	Steps were taken to systematically improve and adapt the project.	↑ 1.44	0.293	0.153	0.377
34	There was ongoing orientation and training available, for example, to new staff or to build capacity.	↗ 0.87	0.034	0.03	0.153	The availability of ongoing training may be critical to whether successful innovations scale up.
35	Staff were given time/incentives to attend the necessary training.	→ 0.47	0.096	0.159	0.178
36	Staff were required to attend the necessary training	→ 0.11	0.348	0.271	0.767
37	The initiative was difficult or complex to deliver.	→ −0.09	0.294	0.163	0.158
38	The initiative helped to make things easier or more efficient.	↗ 0.76	0.86	0.953	0.182
39	The initiative did not require special or extra effort.	↓ −1.09	0.979	0.869	0.597
40	I believe that the staff delivering the initiative found the work/tasks rewarding and satisfying.	↑ 1.59	0.368	0.456	0.743
41	The project team worked well together.	↑ 1.74	0.416	0.796	0.893
42	There were clear responsibilities for individuals the work was shared across the team.	↑ 1.45	0.945	0.533	0.066
43	Project was overly dependent on a particular individual or individuals.	↗ 0.57	0.708	0.355	0.29
44	I believe that the team understood what the project was trying to achieve and that it would lead to improved processes and outcomes.	↑ 1.62	0.218	0.165	0.772
45	There were rewards or incentives that supported engagement with, and continued delivery of, the initiative.	→ 0.07	0.45	0.638	0.228
46	The activities and roles of the initiative were incorporated into job descriptions.	↘ −0.30	0.29	0.243	0.141
47	Staff had time within their working hours to complete the tasks of the initiative.	↗ 0.59	0.251	0.138	0.328
48	The initiative had leadership and/or champions who were committed and capable.	↑ 1.62	0.003	0.001	0.006	Leadership appears to be a highly significant success factor across all definitions of success.
49	There was an appropriate balance of power between those involved with the initiative.	↗ 1.15	0.697	0.775	0.929
50	Team members were able to express their opinions, and their opinions were valued.	↑ 1.91	1 (no variance)	0.049	0.026	Distributed decision-making may be a critical success factor across all definitions of success. It was common to all innovations of value that scaled up (hence no variance).
51	There was a sense of ownership and commitment by those involved with the initiative	↑ 1.79	0.284	0.177	0.102
52	Staff who were responsible for delivering the initiative were involved as partners, and were able to shape the initiative.	↑ 1.74	0.306	0.176	0.031	Participatory processes with staff may be critical to the ability of a project to realise its intended value.
53	The beneficiaries (patients/service users) were involved as partners, and were able to shape the initiative.	↗ 0.83	0.45	0.139	0.027	Participatory processes with patients/service users may be critical to the ability of a project to realise its intended value.
54	The community in which it was situated was involved as partners, and was able to shape the initiative.	↗ 0.96	0.177	0.034	0.023	Participatory processes on a community level may be critical to both the ability of a project to realise its value and its scalability.
55	There was a collaborative network of people/organisations that helped to support and sustain the initiative.	↑ 1.30	0.008	0.007	0.003	The support of a collaborative network of people/organisation may be highly significant to both value creation and scalability.
56	It felt as though the initiative was imposed on us and there was little sense of ownership or commitment to the project.	↓ −1.64	0.684	0.488	0.326

Analysis of variance of potential success factors across categories of success.

*Respondents on average ↑=agree, ↗=somewhat agree, →=neither agree nor disagree, ↘=somewhat disagree, ↓=disagree.

†Asymptotic significances are displayed. The significance level is 0.05. The darker the shading, the safer it is to reject the null hypothesis. Significance <0.05 indicates >95% certainty that the difference between categories is not random.

‡Secondary analysis examined the direction of the association and the strength of effect across categories of success.

Our five stakeholders helped to improve the clarity, acceptability and usability of the survey questions and instructions.

Survey distribution

A neutral administrator from Health Education England distributed the survey by email in August and September 2019 to all 176 named recipients of HIN and Health Education England (South London) funding awards, grants and bursaries. A reminder was distributed 4 weeks later to participants who had not responded. Projects that had received more than one award were sent a single survey, and participants who had run more than one project were sent a separate survey for each project.

Stakeholder follow-up interviews

KLG checked back our results and interpretation with five stakeholders identified by HIN as experienced innovators, one of whom was also involved in the original scoping interviews. Interviews lasted 30–45 min and transcription was facilitated by automated software (otter.ai). These stakeholders helped to refine the model and confirmed its applicability and utility in their context. No new themes arose; however, quotes were used to enrich our survey data.

Data analysis methods

Development of categories of success

KLG and AK categorised projects into grades of success based on how the respondent self-categorised their project, triangulated against their qualitative survey responses. The categories of success were derived through an iterative process, involving both researchers agreeing a descriptive summary of the status of each project (eg, scaled down despite achieving better than expected patient outcomes; scaled down because the intervention did not achieve its aims). We grouped projects with similar project outcomes together, and through a process of constant comparison¹⁴ constructed a categorisation framework that accounted for all the cases in the set.

Determination of salience of success factors

We adopted an exploratory approach to data analysis, which aims to generate rather than test theory.¹⁵ KLG conducted an analysis of variance (ANOVA) for each of the scored factors (Kruskal-Wallis non-parametric ANOVA on rank using IBM SPSS V.25) to see whether there were significant differences between categories of success. The Kruskal-Wallis test does not assume a normal distribution in the data and can be used when the data are ordinal, for example, Likert scores. For asymmetric group sizes, the non-parametric Kruskal-Wallis test performs better than the parametric equivalent ANOVA method.¹⁶

For each factor that was identified as being significantly different between categories of success, we conducted a secondary analysis (box plot for each category) to confirm the direction and consistency of the association. This is generated automatically by SPSS after a Kruskal-Wallis test. A graded ‘exposure-response’ relationship across all grades of success would be expected if a factor genuinely drives success.¹⁷ Where a graded relationship was not present, this is discussed in table 1.

Illumination of success factors

KLG and AK extracted quotes from the survey and interviews relating to each significant success factor to generate a rich description within each theme.

Inductive analysis of limiting factors

KLG coded the content of all qualitative data relating to challenges within projects that had not achieved their intended outcomes or that had scaled down or stopped (n=21) facilitated by NVivo V.12 software. GBR and KLG refined the codes and both authors worked together to inductively arrange the content into themes.¹⁸

Development of final model

We mapped significant factors onto a 2×2 grid using a natural logarithmic scatter plot so that factors that were significant to one dimension of success were mapped to the right half of the grid, factors that were significant to a second dimension of success were mapped to the top half and factors that were significant to both were mapped to the top right quadrant. We grouped success factors into themes through a process of collaborative discussion, and we explored which themes predominated in each quadrant to generate our model which was checked back with stakeholders.

Results

Descriptive summary

We received 63 responses, but seven were incomplete or duplicate so a total of 56 responses (31.8% of 176) were included in the analysis. Each response related to a different innovation project. Survey respondents self-identified within one or more of the following groups: the project leadership team (n=54); service delivery team (n=9); training team (n=9); administrative team (n=6); service lead (n=2); and patient/service user (n=1). Several respondents identified within multiple groups.

Projects were situated in secondary care (n=19); community care (n=14); academic sector (n=5); mental health sector (n=4); online (n=4); primary care (n=3); and the hospice sector (n=2), with the remainder working at the interfaces between services, or across sectors. Their scope ranged from national programmes at hundreds of sites, local programmes supporting tens of thousands of patients, to small intensive innovations working in new ways with a few dozen complex patients, and their duration ranged from 1 to 5 years. The innovation areas related to new ways of working in end-of-life care; disability enablement; support for complex or vulnerable patients; discharge support; pain management; patient safety innovations; recovery and rehabilitation; personalised care; chronic conditions; new models of integrated health and social care; health promotion; and novel simulation and workforce development strategies. Typical projects can be explored at the HIN website¹⁹; however, for reasons of confidentiality, we cannot specify which were included in this study.

Categories of success

Our emergent framework categorised each project’s success across two dimensions: the first relating to whether the innovation was reported as generating more or less than its anticipated value for patients/carers (‘value creation axis’), the second according to whether the project became sustained or scaled up beyond the initial pilot, or whether it was scaled down or stopped (‘expansion axis’). Innovations that were within the scope and intentions of the original pilot were positioned centrally. We initially scored projects into five categorisations across the expansion axis, as some projects expanded locally and some nationally; however, there were not enough projects in each group and statistics became unreliable, so we made a pragmatic decision to adopt relative rather than absolute categories.

The resulting categorical framework is illustrated in figure 3, with the number of innovations in each category shown in brackets.

Figure 3.

Framework for categorisation of success within health service innovation Health service innovations were categorised across two dimensions of success through an inductive classification process. The horizontal axis relates to how successful innovations were in realising their intended value: ‘did it create more or less than its intended value for beneficiaries?’ The vertical axis relates to expansion: ‘was the innovation scaled up or scaled down from the original pilot?’ Numbers in brackets indicate how many innovations were found in each category.

Success factors

Our analysis compared the variance of success factors across innovations that had demonstrated lower than expected value (n=11), value as expected (n=25) and higher than expected value (n=20). Next, we compared variance across innovations that had diminished in scope or stopped (n=17), innovations that were running as expected (n=16) and innovations that had scaled up (n=23). Finally, we excluded low-value projects and analysed again across the expansion axis (n=10, n=12 and n=23, respectively), seeking to explore why innovations with proven value had not been scaled up.

Our analysis is presented in table 1 with significant results (p<0.05) shaded in green. At this level of significance, there is a 1 in 20 probability that a result is in fact random. We have used lighter shading to indicate factors that might potentially be significant, or which could be found not to be significant if the power of the study was increased. The final column gives our interpretation of the more significant findings (p<0.05) that takes into consideration our secondary analysis.

Many factors were similarly scored across all categories of success, for example, information technology (IT) infrastructure. This does not mean that these factors are not important, only that they were experienced similarly across all categories of success and are therefore unlikely to be the underlying cause of the relative success or failure of a project.

We have collated the significant factors together in table 2 with illuminative quotes, and we discuss both positive and negative findings within each of the nine survey themes below.

Table 2.

Factors that are significantly associated with innovation success with illustrative quotes

Significant factors by theme (significant to value or expansion)	Illustrative quotes
Theme 1: Project aims	None of the factors related to the aims of the project were significant.
Theme 2: Resources and support There were sufficient members of staff with the right skills to meet the requirements of the initiative (significant to expansion and value). Members of staff had sufficient energy and time to dedicate to the initiative (expansion). There was sufficient administrative support to deliver and maintain the initiative (expansion). There was sufficient educational support to deliver and maintain the initiative (expansion).	‘The programme support sat with one individual rather than a team and as highlighted previously the administrative/programme support hadn't been entirely appreciated/factored in at the outset of the programme.’ (R16) ‘There are innovators out there who are doing things on their own, and the person I'm thinking about is not in a very good place. He’s got virtually no support, and I don't know how he does it.’ (FI2) ‘The resources needed in terms of administration and support were underestimated. We input far more time and admin resources than originally planned.’ (R15) ‘The envelope of funding available did not enable us to fully develop a training package which was what we had initially hoped to do.’ (R25)
Theme 3: How the project interfaced externally External political or societal factors impacted negatively on the delivery of the initiative (value, scored negatively). We had opportunities to demonstrate the benefits of this innovation within our organisation and/or to other organisations (value).	‘The project piggy-backed on the current social movement highlighting the needs of mental health.’ (R18) ‘Hearing has always been the poor relation to other health issues even though everyone knows someone struggles with their hearing either family or friends.’ (R35)
Theme 4: Organisational factors The initiative integrated well into existing organisational structures, programmes or policies (expansion). The initiative was hampered by opposition from within the host organisation (expansion).	‘The project was presented in [area] Council, [area] NHS Trust, to the public health team in the council and the voluntary sector in [area]. It was aligned with local priorities and local initiatives. A journal article is being drafted.’ (R13) ‘There was also a disconnect between commissioner priorities & [the host organisation’s] priorities in relation to the programme, which impacted on its sustainability & roll-out.’ (R16)
Theme 5: Personal factors I was internally motivated to implement this initiative (value). I found working on the initiative personally rewarding (value). I feel I had the right skills/experience/training to implement and sustain the initiative (expansion and value).	‘I was very motivated to implement this project which was demanding as I had no protected time for it. Nonetheless, you do what is needed to achieve a goal.’ (R18) ‘I am very proud of our achievements and that the work has become a routine part of our culture and system of working.’ (R28) ‘On reflection I needed to lead this project more strongly. I tried to be facilitative and not prescriptive, but the staff were not experienced enough to utilise this opportunity. They needed more direction and support. There was some conflict between the educators and the substantive staff.’ (R10)
Theme 6: Project management The project had sufficient input from experts with the necessary knowledge and experience (value). We were able to demonstrate the effectiveness of the project (value).	‘As the project was run within [university] and [tertiary hospital trust], there was plenty of expertise to call upon as required.’ (R6) ‘The ongoing reporting allowed for the results to be understood early in the project, shared and used in the strategy for delivering education across [region].’ (R5) ‘A robust evaluation was undertaken of the programme, along with regular review points to inform adaptations/opportunities for improvement.’ (R16)
Theme 7: Tasks of the project	No factors relating to the tasks of the project were significant.
Theme 8: Team processes There was ongoing orientation and training available, for example, to new staff or to build capacity (expansion). The initiative had leadership and/or champions who were committed and capable (expansion and value). Team members were able to express their opinions, and their opinions were valued (expansion and value).	‘We always express the value of our service users, administrator, and other members of the team and meet regularly to discuss well-being, progress, and evaluation.’ (R33) ‘The team got on. It was a lovely team and dynamics. We all believed in the idea and were excited about it. Obviously the project needs expertise in [technology], so in that regard the work was dependent on the availability of such expertise within the team.’ ‘Leadership hasn't been invested in providing the platform for the workshops & curriculum to be rolled out. Lip service given by leadership.’ (R56) ‘This programme was carried out by a team but lead by myself. Other parties lacked the time and incentive to commit to running and leading the programme after the 12 months of my time being project lead.’ (R46)
Theme 9: Collaborative and participatory practices Staff who were responsible for delivering the initiative were involved as partners, and were able to shape the initiative (value). The beneficiaries (patients/service users) were involved as partners, and were able to shape the initiative (value). The community in which it was situated was involved as partners, and was able to shape the initiative (value). There was a collaborative network of people/organisations that helped to support and sustain the initiative (expansion and value).	‘We've had some sort of service user involvement all the way through… that’s really improved the way we’ve reflected and talked… it gives a genuineness to what we're trying to achieve… the fact that you go and work with the associates and carers, you actually go and look at the environment, you spend time with the nurses that you’re going to be teaching and all the other people that you’re working with, really helps to, you know, it definitely adds value to the project.’ (FI3) ‘There was a single practitioner using the resource on her own, and now it is nationally recognised… Without that level of support: the time, the people and the resources, we wouldn’t have got anywhere. It’s been a fantastic level of support. They designed an implementation toolkit to support practitioners embedding the programme locally.’ (FI2) ‘The team at the [innovation network] were fabulous and so supportive.’ (R12)

The nine themes and factors were derived from a qualitative review of the literature supplemented by stakeholder interviews. Significant factors were derived through a survey that explored salience of factors to outcomes. Survey respondents are indicated by R(n) and follow-up interview respondents by FI(n).

Project-related factors

Interestingly, the aims of the project did not appear to be critical to success. Both successful and unsuccessful innovations were similarly reported as being designed to address an important healthcare need that was concerning to the public. All funded projects were required to articulate a credible evidence base arguing that stated benefits could be achieved through the project plan.

Resourcing and expertise

All significant resource-related success factors were associated with the workforce. Non-critical factors included infrastructure (such as buildings, materials and supplies), which were reported as sufficient; IT, which was moderately good across all categories of success; and funding issues, which were also similar across all categories of success. However, having the right numbers of staff with the right skills appeared to be highly significant, both in terms of the project being able to realise its intended value, as well as for it to become scaled up beyond the original pilot. Staff with time and energy appeared critical to whether successful innovations became scaled up, as were administrative and educational support, including the availability of ongoing educational support (eg, orientation and training for new staff, or to build capacity). Expertise appeared to be critical across all categories of success, both in terms of the innovator feeling they had the right skills, experience or training; the project having access to staff with the right skills; and having external expert input where needed.

External factors

Alignment to societal needs appeared to correlate with whether a project was able to realise its intended value, but less so with its expansion. However, as the effect size did not grow consistently across categories of value creation, we cannot necessarily infer a causal relationship.¹⁷ Qualitative comments indicated that projects that were able to align themselves to current political or societal agendas, such as mental health, were more successful. Conversely, those attempting to work in relatively less topical areas of practice described difficulty securing strategic funding, so we have tentatively included this factor in our model.

Organisational factors

Our analysis of organisational factors indicated that the ability of an innovation to integrate into existing organisational structures, programmes or policies may be critical to whether it scales up, and possibly also to its ability to create value (p=0.059). Successful projects described adapting where necessary to achieve a good fit within organisational priorities. For the most part, host organisations were described as having a positive learning culture and were ready and able to undertake innovative initiatives; however, even innovations with proven value were unable to survive if there was opposition within the host organisation.

Personal factors

Few respondents reported being released from other duties so that they could implement their initiative. However, most respondents said they benefited from a supportive peer culture. Respondents who were able to realise value were significantly more like to say they were internally motivated and found working on the project rewarding.

Project management

Most projects measured or assessed the outcomes and impacts of the project, though this appeared to be more common in successful projects (p=0.060). Projects with high value were able to demonstrate and share this success. Leadership appeared to be a highly significant success factor across all categories of success, with struggling or unsuccessful projects citing leadership failures.

The tasks of the project

Similar to theme 2, which explored the aims of the project, the tasks of the project did not appear to be significantly different across categories of success.

Collaborative and participatory practices

Valuing team members’ opinions was highly significant across all categories of success and was present in all projects that were scaled up (hence variance not calculable). Participatory approaches were significantly associated with the ability of an innovation to generate value. These participatory processes related to the staff delivering the innovation, the intended beneficiaries and the communities in which the innovations were situated.

Finally, one of the most significant differentiating factors across all categories of success was engagement with a collaborative network that helped to support and sustain the initiative.

Limiting factor analysis

In addition to the above success factors, which were quantitatively identified, the following limiting factors were identified through our qualitative analysis of failed or struggling projects. As our limiting factor analysis is qualitative and interpretive, we present our data in line with our analysis.

Boundaries between commercial, voluntary and public sectors

While UK healthcare is primarily publicly funded and provided by the NHS, social care is often commercially provided,²⁰ creating the potential for friction at the interfaces between these sectors.

As the care homes are private businesses, there was some lack of political will to embrace the training, as there was a view that although there was the potential to improve health outcomes for the residents, the manager did not feel there were sufficient resources to implement the required training. (R4)

Commercial organisations were reported as unwilling to release staff for training unless the value of that training was felt within the organisation. Valuable initiatives by the voluntary sector to train social care staff, but which provided benefits in the healthcare sector, fell between sectors and were potentially unviable without direct funding.

The voluntary sector is happy to participate but there is no spare capacity within it unless there is a financial package that can go with it. (R35)

There was concern that privately funded organisations were not subject to the same standards and mandates as the publicly funded bodies, and were failing to invest in training.

Because it is not mandated, organisations do not have to engage with or release staff for education. (R2)

Restructuring within the NHS has created a set of semiautonomous institutions and organisations with different and sometimes competing priorities.²¹ Some participants described difficulty aligning project aims to multiple organisational goals.

There were tensions between the two boroughs in relation to approach & resourcing. There was also a tension between commissioner expectations and practice/federation expectations which have impacted on the programmes sustainability. (R16)

So, this intervention has a good return on investment, for every £1 you spend you get a return of £5.20. And they’ll say, I’m the one making the investment, but he’s the one making the return here. I’ve got a budget; he’s got another budget. We might both be in the health system, but I’m not going to spend my money if he’s the one getting the return. (FI2)

Workplace cultures and priorities

Some projects reported finding non-healthcare workers receptive to health-related training; however, some failed or struggling projects found this a challenge.

Medicines delivery teams unwilling to take on additional role. (R61)

There were concerns raised by care home managers that the initiative would cause undue responsibility on individuals to make clinical decisions. (R16)

Participants described differences between academic and workplace learning cultures, and variable receptiveness of front-line clinical staff to change. Some described resistance to outsiders telling healthcare workers how to improve. This may reflect the inverse of high-value projects, which were found to engage in participatory practices, engaging patients, front-line staff and communities in codesigning their innovation.

It has been difficult to embed these products due to structural issues within the staff teams. (Nursing) It was clearly not seen as a priority. (R20)

I think the main insight I would have is that when working with mental health nursing teams the researcher and research team needs to be fully integrated into team(s) and seen as part of the culture. Being an outsider does not seem to work as day-to-day practice seems to regulate research. (R20)

Participants also described tensions between management priorities and the priorities of those working directly with patients.

No interest on part of management. I don’t think they have even read it. (R57)

The initiative was welcomed at service level, however there was little interest at senior management level. (R52)

There is such a dislocation between commissioning and what is happening on the ground. (FI2)

Lack of support beyond the start-up phase

Participants noted ongoing privileging of new innovation over sustaining or scaling up innovations that have already demonstrated their value. For example, clinical academics do not gain publications for ongoing maintenance of innovative practice:

‘Research remit probably wouldn’t cover [further dissemination] unless there was a good likelihood of REFability’ [‘REF’ refers to the Research Excellence Framework, a scoring system used to fund the university sector]. (R1)

Participants described innovation funding streams, but articulated difficulty securing funding beyond the start-up phase.

The project was resourced sufficiently for the pilot. However, once the pilot finished so did the project. (R5)

The education faculty and funding is driven towards innovation and not sustainability—this de-incentives individuals from continuing with existing projects. (R8)

Burn-out, turnover and lack of protected time

Participants described projects that were limited by staff burn-out, turnover and a lack of protected time.

My commitment to the project was there however the resources I had to continue with project were limited due to competing pressures on my time. (R12)

The programme required more administrative support than anticipated & this ended up being an ask over & above someone’s day job for a prolonged period of time. (R16)

The most important person was our pharmacist who moved from the pharmacy a few months after we started! (R61)

Risk as integral to innovation

Finally, it is worth noting that participants felt that risk was a necessary ingredient of healthcare innovation. Innovations that fail to demonstrate value should be supported in folding without hesitation, and lessons shared.

The project demonstrated that this initiative was not a model that would work in the hospital environment hence could not be embedded. (R30)

Final framework

We created two 2×2 matrices containing all the significant success factors across each dimension of success, shown in figure 4. The matrix on the right excluded low-value projects in the calculation of factors significant to expansion and served to support the inclusion of some marginal factors in the final model as they became more significant despite lower power.

Figure 4.

Critical success factors plotted according to their salience to success. Critical success factors plotted according to their significance to success on a natural logarithmic scale so that factors above and to the right are probably significant (p≤0.05). The expansion axis indicates significance to whether a project is scaled up or down beyond the original pilot. The value creation axis indicates significance as to whether it creates more or less than its intended value for beneficiaries.

Figure 4 shows clearly congruent clusters of factors in each quadrant, indicating that some types of factors may be more important to expansion, while others are more important to value creation. These clusters relate to skills and expertise, leadership and motivation, organisational fit and structural support, societal alignment and participation, and evaluation.

As outlined in table 1, there are questions as to whether evaluation and motivation are dependent rather than independent variables: does finding working on a project personally rewarding drive success or vice versa, and does a positive evaluation drive success or vice versa? Triangulation with qualitative comments (in table 2) suggests that evaluation and motivation may drive success, so they have been tentatively included in our final model.

Themes that were predominantly related to value creation (participation, motivation and evaluation) were labelled value creation factors. Themes that were predominantly significant to expansion (organisational fit and structural support) were labelled expansion factors. Themes that were significant to both axes (expertise, leadership and a supportive network) were labelled core success factors. We arranged success factors into a nested hierarchy, as innovations that do not generate value are unlikely to be scaled up. Our final model in figure 5 also lists potential limiting factors identified through our inductive qualitative analysis.

Figure 5.

Nested hierarchy of success factors and limiting factors for health service innovation. Factors that may be significant to both value generation for the intended beneficiaries and to whether the innovation is scaled up beyond the original pilot are labelled as core needs. Factors that may be significant specifically to value generation are the next priority (middle layer) as innovations that do not generate value will not become embedded or spread. Finally, factors that may primarily be significant to expansion are presented in the outermost layer. Additional limiting factors were identified through an inductive analysis of qualitative responses from projects that had scaled down or failed to produce their intended value.

Discussion

This analysis of 56 health service innovation projects has enabled us to propose a model for understanding success in health service innovation that has two discrete axes: one relating to whether or not the innovation created value for its intended beneficiaries; the other relating to whether or not it was scaled up beyond the original pilot. Comparing projects across these dimensions of success has enabled us to hypothesise that:

• The core drivers of success are leadership and collaborative expertise (leadership skills and commitment, expert input, sufficient staff with the right skills and expertise, and a supportive collaborative network).

• The drivers of value creation for the intended beneficiaries are participation, motivation and evaluation (involvement of patients, public, practitioners and communities, alignment to societal needs, internal motivation, finding the project work rewarding, ability to demonstrate benefits and having opportunities to share impacts).

• The drivers of sustainability and scale-up are organisation fit and structural support (organisational fit and alignment, administrative and educational support, staff with time and energy).

Additional limiting factors included difficulties at the boundaries and intersections between organisations, professions, sectors and cultures; a lack of structural support beyond the start-up phase; and staff burn-out and turnover.

Within healthcare services, the issue of diffusion and sustainability of innovation has received widespread academic attention pioneered by Greenhalgh et al²² who drew on Rogers’ seminal text on diffusion of innovations.²³ There have been many subsequent notable academic contributions.^24–28 Nilsen proposed an overarching framework of healthcare implementation theories according to the aim of the theory.²⁹ Theories such as those about innovation sustainability, which include the diffusion of innovation theory, were categorised as ‘determinant frameworks’, as they posit general types of factors that can influence the success of an innovation. We believe that our findings contribute through empirical evidence to theoretical development at this level, and thus may have wider implications than programme-level data would normally allow. According to Nilsen, such theories have typically been analysed and formulated across individual studies, at the level of meta-analysis or review²⁹ and may therefore be one or more steps removed from the underlying data. This study is different in that we have developed mid-range theory that is empirically grounded in programme-level data, and there is a clear line between our data and the generated theory.

Conceptions of innovation success tend to focus on sustainability³⁰ and scale-up.²⁴ We suggest that both are contingent on the ability of an innovation to provide value to its intended beneficiaries in the first place. There are few theories grounded in empirical data that explain this dimension of success. Our findings highlight the importance of patient, public and practitioner involvement, alongside the core success factors of leadership and collaborative expertise. We suggest that these are fundamental preceding factors to either sustainability or scale-up.

This ‘value for intended beneficiaries’ dimension of success also allows us to conceptualise a valuable innovation that is not growing or expanding. This, we argue, is important: healthcare innovations may have parameters within which growth and expansion are constrained, perhaps because their aims have been achieved, or because the context changes. An innovation that has met its aims but has not expanded beyond its natural boundary should be properly positioned as such.

Our ability to research a set of potentially valuable projects that were scaled down or stopped, many of which never reach the literature, may have afforded novel insights. Fixsen et al suggest that sustainability can only be asserted when the funding to support implementation is withdrawn.³¹ Wiltsey Stirman et al’s systematic review suggests sustainability can be asserted after a period of 2 years.³² Our findings suggest that continued structural support, particularly organisational, administrative and educational support, may be critical to a project’s sustainability and scalability, and that their withdrawal may destroy potentially valuable innovations.

Finally, our findings further validate the work of Dopson et al,³³ whose qualitative exploration of a similar set of health service innovations highlighted the importance of context and process over content: it is not so much what you are trying to achieve, it is how you do it and the organisational and interpersonal contexts that you work within that matter.

A limitation of this research is its highly contextual nature. Our results may not be generalisable to all contexts; however, repeating these methods may produce locally relevant results. The ANOVA depends on the universe of potential factors having been correctly identified and a large enough number of innovations to produce statistical significance. The research could be improved by more extensive validation of factors, patient and public involvement, further testing the directionality of tentative factors, a greater geographical spread and a greater number of projects to allow for finer grading across the expansion axis.

Conclusions

Our findings suggest that organisations and policy makers wishing to support service-level innovation in similar healthcare contexts address the factors identified through this research as critical to success.

Such strategies might include:

• Supporting innovators with the right skills and expertise, including leadership skills, implementation support and evaluation expertise.

• Innovation networks to provide opportunities to showcase success and provide a peer community of expertise and support.

• Emphasising participatory practices and collaborative approaches, so that innovations are more likely to align to societal and organisation goals and generate value for patients, communities and practitioners.

• Providing administrative and educational support during the scale-up phase, and ensuring that this support is maintained or handed over rather than withdrawn to schedule.

• Recognising and enhancing the internal motivation and drive of innovators as well as more goal-oriented motivations such as career needs.

At a structural level, the boundaries between organisations, professions and the health and social care sectors may need to be addressed as potential barriers to successful innovation.

More research is needed to confirm whether addressing these factors prospectively enhances the success of future innovations.

Data availability statement

Data are available upon reasonable request. Due to the highly individual nature of healthcare innovations and the limited geographic area of this study, we are unable to provide our raw data. We undertake to provide a redacted data set upon reasonable request.

Ethics statements

Patient consent for publication

Not required.

Ethics approval

Ethical approval was granted on 26 March 2019 by the Research Ethics Committee of King’s College London (LRS-18/19-10432).