Factors predicting timely implementation of radiotherapy innovations: the first model

Abstract

Objective:

The improvement of radiotherapy depends largely on the implementation of innovations, of which effectivity varies widely. The aim of this study is to develop a prediction model for successful innovation implementation in radiotherapy to improve effective management of innovation projects.

Methods:

A literature review was performed to identify success factors for innovation implementation. Subsequently, in two large academic radiotherapy centres in the Netherlands, an inventory was made of all innovation projects executed between 2011 and 2017. Semi-structured interviews were performed to record the presence/absence of the success factors found in the review for each project. Successful implementation was defined as timely implementation, yes/no. Cross-tables, Χ² tests, t-tests and Benjamin-Hochberg correction were used for analysing the data. A multivariate logistic regression technique was used to build a prediction model.

Results:

From the 163 identified innovation projects, only 54% were successfully implemented. We found 31 success factors in literature of which 14 were significantly related to successful implementation in the innovation projects in our study. The prediction model contained the following determinants: (1) sufficient and competent employees, (2) complexity, (3) understanding/awareness of the project goals and process by employees, (4) feasibility and desirability. The area Under the curve (AUC) of the prediction model was 0.86 (0.8–0.92, 95% CI).

Conclusion:

A prediction model was developed for successful implementation of innovation in radiotherapy.

Advances in knowledge:

This prediction model is the first of its kind and, after external validation, could be widely applicable to predict the timely implementation of radiotherapy innovations.

Introduction

Innovation implementation in radiotherapy centres aims at continuously improving patient outcomes at affordable costs.^1,2 Innovation can be defined as “The intentional introduction and application within a role, group or organisation of ideas, processes, products or procedures new to the relevant unit of adoption, designed to significantly benefit the individual, group or wider society”. This is a general accepted definition among researchers.^3–5 Earlier research showed that in The Netherlands, the number of successfully implemented innovations in radiotherapy varies largely, indicating there is room for improvement.⁶ To realise the latter, we need to get more insight in determinants of successful innovation implementation. There are no radiotherapy-specific studies in this area but in general, there is a lot of research conducted into these success factors and innovation implementation frameworks. However, it is difficult to determine the coherence of these factors and frameworks, and which factor is more important than others. In this study, we want to address this problem, by developing a prediction model that contains only those statistically significant determinants that predict the chance of successful innovation implementation.^7,8 To date, we are not aware of a predicting model that can help organisations, very specifically, to improve innovation implementation.

Therefore, the aims of this study are:

1. To identify success factors, i.e. factors that are associated with improved innovation implementation in clinical routine in radiotherapy.

2. To build an internally validated model, based on these identified success factors, to predict the chance on successful innovation implementation in radiotherapy.

Methods and materials

Literature study on success factors of innovation implementation

A literature study on reviews was carried out, because reviews give a comprehensive overview of success factors attributing to effective innovation implementation. The databases PubMed and Web of Science were screened for relevant publications in English with an abstract including the terms stated in Supplementary Table 1, in the period from 2009 to June 2018. Next to the search strategy, the following categories were included in Web of Science: health care sciences, nursing, computer science interdisciplinary applications, management, medical informatics, radiology nuclear medicine medical imaging, planning development, business, computer science information systems, computer science artificial intelligence, social issues, economics, communication, business finance, operation research management science, multidisciplinary sciences and oncology.

The publications retrieved with the search strategy were screened according to the criteria defined in Supplementary Table 2 by two researchers, independent of each other. Initial disagreements on study selection were discussed until consensus was reached. When the screening of the criteria from the abstract was inconclusive, the full text publications was assessed. After assessment, the selected publications were used to make an inventory of all mentioned success factors.

Data collection

Inventory of innovations

First, an inventory was made of all innovation projects from the period 2011–2017 in two large independent radiotherapy treatment centres in the Netherlands. Furthermore, two independent researchers, not involved in the data analysis, studied policy plans, memos and relevant written documents and analysed these documents to complete the list of innovations. Projects were assigned by the investigators, to different innovation types, namely product (or treatment) innovation, technological innovation or organisational innovation.⁶

Framework to categorise the size of innovations

Subsequently, a list of criteria was used in order to be able to categorise various innovations according to the project size. Three categories were defined, namely the number of hours budgeted for the project, the number of disciplines needed and the number of project members needed.

The above-mentioned categories for project size are defined in Table 3. All projects have been scored on these criteria, through semi-structured interviews, and have been assigned points. Subsequently, an aggregated measure has been developed, which is the average score on the project size criteria. For example: a project with ≥1 FTE (three points), two disciplines (two points) and three employees (two points)=7/3=2.3= medium project (Table 1).

Table 1.

Criteria to determine the size of project

Criteria project size	Number of hours budgeted for the project	Disciplines needed	Project members needed	Aggregated measure for project size
Small	0–0.5 FTE or 800 h (one point)	1 (1 point)	1–2 (one point)	1 (=3–4.5 points/3)
Medium	0.5–1.0 FTE equal to max 1600 h (two points)	2–3 (two points)	3–4 (two points)	2 (=4.6–7.5 points/3)
Large	>1.0 FTE or >1600 h (three points)	>3 (3 points)	>4 (3 points)	3 (=7.6–9 points/36/)

Definition successful implementation

Innovation implementation was considered successful if the implementation was completed within ultimately 6 months after the planned end date.

Identifying success factors for all innovations

Semi-structured interviews (for the questions see Supplementary Table 3) with either the head of the department, the head of physics or the project leaders were scheduled, to score the various perceived success factors for all innovations as yes/no/not applicable and whether the innovation was successful (scored as yes/no/yes with delay). In total 40 people were interviewed. Construct validity was given substance by operationalising all factors such as “teamwork” and “positive attitude” with several examples and specified questions (Supplementary Table 3).

Data analysis

The relationship between various success factors for innovation implementation and the success of implementation was analysed, for each size and type of innovation. In addition, the relationship between the (type of) innovation and the successfulness of the innovation was analysed. It was also analysed whether or not there were any differences between the two centres in innovation type, project size and success factors.

Statistics

First univariate analysis of all data was performed; subsequently, multivariate analysis was performed to develop a prediction model.⁹ Internal validation was carried out, and a nomogram was developed.¹⁰ All analyses were performed using SPSS for Windows v. 25 with the exception of the internal validation which was carried out using R v. 4.0.1 (6 June 2020) and package rms v. 6.0–0 (4 June 2020). The receiving operator characteristic (ROC)-curve was built by calculating the probability of success for each combination of determinants in the multivariable logistic model and subsequently determining sensitivity and specificity for each cut-point.

The univariate associations between success factors and implementation of innovations were analysed using Χ² tests. For all success factors that were significantly associated with successful implementation, a check was performed to determine which success factors were closely associated, in order to prevent multicollinearity in case of inclusion in a multivariate model. Also a Benjamin–Hochberg correction was performed. For all tests, the significance level was set at p < 0.05.

A forward stepwise multivariate logistic regression analysis was done to select the combination of success factors that have prognostic value for implementation of innovations. The initial model included only the innovation type. Factors were added to the model one at a time, starting with the factor that was univariate the most statistically significantly associated. Improvement of the model was determined from changes in the −2LogLikelihood (−2LL) value. When the model did not improve anymore, no more factors were added. The final model was used to calculate the probability of success for each innovation and accuracy of the model was determined using the area under the receiving operator characteristics curve (AUC). An internal validation with correction for optimism was calculated using a 100 times bootstrap procedure. Finally, a visual representation of the model in the shape of a nomogram was made.

Results

Success factors innovation implementation found in the literature review

The search identified 189 unique articles (Figure 1), of which 19 were included in this study. We found eight main categories of success factors in literature, namely workload, leadership, integration research/clinician, organisational culture, teamwork, perceived benefits of the innovation, compatibility & complexity and external barriers. These main categories contain 31 success factors. Supplementary Table 3 gives an overview of the 8 main categories and the 31 proven criteria and the question asked during the interview.

Figure 1.

PRISMA diagram of the articles included in the literature review.

Number of implemented innovations in the two radiotherapy centres and the success rate

A total of 163 innovation projects [45% technological, 22% organisational and 33% treatment innovations (Supplementary Table 4 gives an example of innovations per type)] were included in the study, of which 54% were successfully implemented as planned. Table 2 lists the type of innovations included and if the innovations were implemented or not.

Table 2.

Type of innovations sorted according to the implementation status, successfully implemented or delayed/not implemented

Type of innovation	Successfully implemented	Delayed or not implemented	Total
Technological	46	27	73
Treatment (product)	20	34	54
Organisational	22	14	36
Total	88	75	163

When looking at type of innovation project, one centre had almost no organisational innovations, in this comparison organisational innovations were excluded. Comparing the two centres on treatment and technological innovations no significant differences were found in innovation type and project size. Using multivariate analysis, organisation was added to the model. When adding organisation to the model no, it did not add predictive value to the model.

Relation between successful innovation implementation and the type or size of the project

A significant relation was found in univariate analyses between type of innovation and the success of innovation implementation (Χ² p = 0.009). Technological innovations were used as a reference. Univariate, organisational innovations were 1.45 more likely successful, and treatment innovations were 0.35 less likely successful.

Project size (aggregated measure) had a significant correlation with innovation implementation (p = 0.023). Small projects were used as a reference for all three criteria. For number of hours budgeted for the project, a medium project was 0.59 times less likely successful than a small project, whereas large projects were 0.46 times less likely successful. For number of disciplines needed, medium projects were 2.83 times more likely successful than small projects, whereas large projects were 0.33 times less likely successful. With respect to number of project members, medium projects were 1.39 more likely successful (OR = 1.39) and large projects were 0.93 less likely successful (OR = 0.93), (Χ² p value of p = 0.404).

Success factors related to successful innovation implementation per innovation type?

(Table 3) shows that 14 of the 31 success factors found in literature, showed a significant impact on successful implementation in the univariate analysis. When studying these 14 success factors more closely per innovation type some significant differences per type were found: Integration between research/clinician was significantly less present in organisational innovations compared to both technological and treatment innovations. Positive organisational climate and clearly identifying the role, actions and importance of each member’s role for each step of the procedure were significantly less present in organisational innovations compared to technological innovations. Feasibility & desirability was significantly better/more present in technological innovations compared to both organisational and treatment innovations. In organisational innovation, there were significantly more ‘other’ factors than the above-mentioned present compared to technological innovations (Table 4).

Table 3.

Success factors with a significant correlation with innovation implementation in radiotherapy in the univariate analysis

Main categories of success factors	Success factors	Odds ratio for successful implementation	95% Confidence interval	p value	Benjamini–Hochberg significance
Workload	Sufficient resources	4.51	3.67–5.36	<0.001	0.001
	Sufficient and competent employees who are satisfied with the terms of employment and development opportunities	9.25	8.13–10.36	<0.001	<0.001
Leadership	Supporting leadership	2.45	1.76–3.13	0.011	0.025
	Effective project management	3.31	2.63–3.98	0.001	0.003
Integration research/clinician	Integration research/clinician	0.43	−0.21–1.07	0.011	0.025
Organisational culture	Positive organisational climate, friendly and respectful climate	7.82	6.54–9.09	<0.001	0.001
	Organisational readiness for change and adapt if needed	2.87	1.96–3.78	0.028	0.053 (ns)
Teamwork	Teamwork and participation	6.26	5.12–7.40	0.001	0.003
	Clearly identifying the role, actions and importance of each member’s role for each step of the procedure	5.09	4.17–6.01	<0.001	0.002
	Understanding and awareness of aims and process of the project	7.31	6.36–8.27	<0.001	<0.001
	Training and awareness campaigns	2.03	1.38–2.68	0.034	0.062 (ns)
Perceived benefits	Applicability, perceived benefits	18.11	16.05–20.18	<0.001	0.001
	Feasibility and desirability	14.68	13.19–16.18	<0.001	<0.001
Compatibility & complexity	Complexity (integration of functionalities and devices/many professionals)	0.19	−0.50–0.88	<0.001	<0.001
External barriers	Market factors, government regulations, and technological limitations	0.37	−0.42–1.17	0.018	0.038
	Other	0.14	−0.77–1.05	<0.001	<0.001

Table 4.

Significant differences in presence of success factors between the different types of innovations

	Success factor applicable	Integration research/clinicians	Positive organisational climate, friendly and respectful climate	Clearly identifying the role, actions and importance of each member’s role for each step of the procedure	Feasibility & desirability	Other
Technological innovation	Yes	36 (49%)	68 (93%)	65 (89%)	70 (96%)	9 (12%)
	No/Not applicable	34 (47%)	4 (6%)	7 (10%)	2 (3%)	60 (82%)
	Missing	3 (4%)	1 (1%)	1 (1%)	1 (1%)	4 (6%)
	Total	73	73	73	73	73
Treatment innovation	Yes	29 (54%)	48 (89%)	42 (78%)	42 (78%)	12 (22%)
	No/Not applicable	25 (46%)	6 (11%)	11 (20%)	11 (20%)	38 (70%)
	Missing	0	0	1 (2%)	1 (2%)	4 (8%)
	Total	54	54	54	54	54
Organisational innovation	Yes	6 (17%)	27 (75%)	24 (67%)	28 (78%)	14 (39%)
	No/Not applicable	30 (83%)	9 (25%)	11 (30%)	8 (22%)	21 (58%)
	Missing	0	0	1 (3%)	0	1 (3%)
	Total	36	36	36	36	36
Overall p value (Pearsons Χ2)		0.001	0.012	0.020	0.002	0.008
Technological vs Organisational innovation p value (Pearsons Χ2)		0.001	0.003	0.005	0.001	0.002
Technological vs Treatment innovation p value (Pearsons Χ2)		ns	ns	ns	0.001	ns
Organisational vs Treatment innovation p value (Pearsons Χ2)		<0.001	ns	ns	ns	ns

ns, Not significant.

Prediction model

The final prediction model contained the following determinants: (1) sufficient and competent employees, (2) complexity, (3) understanding/awareness of the project goals and process by employees, (4) feasibility and desirability (Table 5). In Figure 2, the prediction model is shown as a nomogram.

Table 5.

Determinants included in prediction model

Determinants	Regression weight (B)	SE	Odds ratio (95% confidence interval)	p value
Technological innovation type				0.001
Organisational innovation typea	1	0.58	2.0 (0.65–6.2)	0.230
Treatment innovation typea	−1.44	0.50	0.24 (0.09–0.62)	0.004
Sufficient and competent employees who are satisfied with the terms of employment and development opportunities	1.86	0.72	6.45 (1.57–26.42)	0.010
Complexity (variety of functionalities and devices/many disciplines involved)	−1.73	0.46	0.18 (0.07–0.44)	0.000
Good understanding and awareness of the goals of the project and the process of implementation	1.57	0.59	4.8 (1.52–15.14)	0.007
Good feasibility and desirability	1.65	0.90	5.22 (0.90–30.37)	0.066
Constant	−2.90	1.26	0.06	0.021

SE, Standard error.

^aCompared to technological innovation type.

Figure 2.

Nomogram to estimate the probability of successful implementation, based upon different characteristic of an innovation project.

Apart from the above-mentioned determinants, type of innovation was also included in the prediction model. When studying the four success factors more closely, multivariate and corrected for innovation type, no significant differences were found between technological and organisational innovations. This was different for treatment innovations. For all four success factors, in multivariate analyses, a significant difference was found between technological innovations and treatment innovations. In the analyses technological innovations were used as a reference. We found in our data that treatment innovations have four times less likely a successful implementation as technological innovations (p = 0.004) had. The size of the project was not a factor that made the model stronger.

The AUC of the prediction model was 0.86 (0.8–0.92, 95% confidence interval) (Figure 3), this figure is pre-bootstrapping because the added value of a post bootstrapping figure seemed to be neglectable. Corrected for optimism, the AUC was 0.83 (95% CI 0.78–0.91).

Figure 3.

AUC to indicate the accuracy of the nomogram. AUC, Area under the curve, ROC, Receiver operating characteristic.

Discussion

We were able to build a prediction model that can help radiotherapy centres to calculate the expected successfulness of an innovation implementation before the project starts. On this basis, it is possible to manage those factors that cause a low score in the prediction model in such a way that the chance of successful implementation increases.

The relevance of success factors with a significant impact on innovation implementation

Apart from the four success factors in the model, we found several other factors that were significantly related to successful innovation implementation in the univariate analysis (Table 5). However, we found significant differences in the presence of these success factors between the innovation types. We cannot explain these differences very well on the base of previous literature, except for the factor “integration research/clinician”. This factor was less present for organisational innovation, because in the most organisational innovations researchers are not involved. We need to perform further research to address the substantiation of arguments for the found differences per innovation type.

The fact that only four factors were included in the model does not mean that the other factors are not related to innovation implementation. Timely innovation implementation always depends on multifactorial causes. However, for the prediction of timely implementation limited factors are relevant.

As shown in the results, the determinants predicting success in the final nomogram were complexity, sufficient and competent employees, understanding/awareness of the project goals and process by employees and feasibility and desirability, and type of innovation. The nomogram was primarily developed to identify factors that can easily be managed to improve the probability of successful implementation. Innovation type and complexity are however difficult, or impossible, to manage: type of innovation is a given condition; an innovation is either a technological, treatment or organisational innovation. We found that treatment innovations were four times less likely successful regarding implementation than technological innovations. . More research is required to investigate possible causes. As mentioned above, complexity is difficult to manage as well. In this research, complexity was defined by many functionalities, many devices and many professionals. For complexity the odds ratio (OR) is 0.18. This implies that when implementing a complex project, the chance of success is five times lower than a project that is not complex. Although complexity is difficult to reduce, in literature is stated that by drawing on various sources for ideas and information about implementation and by creating work conditions that encourage personal and team creativity the project leader can reduce the risk of failure of implementation of complicated projects.¹¹ Furthermore, complexity can also possibly be reduced by dividing a project in to smaller subprojects. Nevertheless, it is important to recognise that these difficult to manage factors do influence the chance on successful implementation, (1) to manage expectations, (2) to make decisions on whether to continue or not; (3) to take into account additional time (or maybe resources) to enhance successful implementation.

Predictive factors that can more easily be managed are (a) sufficient and competent employees, (b) understanding/awareness of the project goals and process by employees and (c) feasibility and desirability.

(a) Management must ensure that, at the start of a project, there are sufficient and well-trained employees to make the project a success. Sometimes, the eagerness to start is so high that a project is started without ensuring that there are sufficient employees to work on the project. This however, often results in frustration, since successful implementation is not reached. When there are sufficient and competent employees participating in an innovation project and those employees are satisfied with the terms of employment and development opportunities the OR is 6.45. This means that the chance of successful implementation is 6.45 times higher than for projects that are lacking resources. Obviously, sufficient and competent employees can find the means to overcome the barriers associated with the innovation and ensure the benefits of the innovation.¹¹

(b) Understanding/awareness of the project goals and process by employees is key to ensuring effective innovation implementation. It is important that employees understand the process and goals of a project, this leads to long-term positive outcomes and reduction in overall burden.¹² If there is good understanding and awareness of the goals of the project and the process of implementation the chance of successful innovation implementation is almost five times bigger than when there is no understanding and awareness (OR is 4.80). Insufficient understanding/awareness of the project goals and process can trigger unnecessary resistance toward the implementation.¹² Lack of awareness can be addressed via interactive communication, additional training and education.^12–14

(c) Other important factors are feasibility and desirability. Interaction plays an important role in enhancing feasibility & desirability. People (professionals and patients) hear about, and use innovations through interaction. Good interaction and communication and facilitating people to develop the necessary skills may enhance the desirability of innovations.^15,16 Especially, medical specialists’ scientific, clinical and social arguments for or against a certain innovation, and his/her knowledge and vision of the context in which an innovation is used, define the desirability of this innovation. It is the medical specialist who sets the stage for the routine use of innovations. Their expertise, social status and the power of their profession ensure that they play an important role in the evaluation of innovations and their desirability.¹⁷

According to literature also, feasibility from the perspective of the organisation and the patients seems to be relevant.¹⁸ In our study, good feasibility and desirability increased the chance of successfully implementing an innovation by more than five times (OR found is 5.20).

When managing the above-mentioned factors well, the chance of successfully implementing an innovation is likely to increase.

How to use the prediction model in clinical practice

To make the prediction model practical, we have written a standard operation procedure (SOP) on how management can use the model in clinical practice after external validation (Figure 4). This SOP sheds light on the combined factors of importance and enables health-care institutes to manage factors of importance and improve the outcome of innovation implementation, or to manage expectations. The SOP can be used by letting all project members score the project/innovation before the start of a project. If the chance of success is not higher than a certain percentage, to be defined by the organisation, it is not wise to start the project and the project leader must first improve the insufficient implementation conditions.

Figure 4.

SOP, Standard operating procedure.

Strengths and limitations

One of the strengths of the research is that it is not solely based on data from one radiotherapy centre but that two large academic centres with many innovation implementations participated in the research. This also immediately leads to a limitation of the research. An important step was taken to build an internally validated prediction model. External validation is necessary for using the model in practice. Furthermore, the model was fitted and assessed for prediction value using the same data set, which limits the validity of the prediction model. This might raise some concerns about overfitting and the presence of multiple hypothesis testing. Due to the retrospective nature of our study, it might be subject to hindsight bias. In addition, in this research inter-rater bias cannot be completely excluded. Some of the success factors included in the model have fairly wide confidence intervals around the OR. In subsequent research, we will test the validity of the model in a third radiotherapy centre as well as in another medical specialism in health care. Another limitation is that success factors were scored dichotomous. However, we thought it would be wise to use dichotomous instead of continuous scales because, otherwise, we estimated that a lot of scores would be put around the average, not resulting in a clear picture what factors really matter. Finally, a limitation is that success factors were scored retrospectively. Ideally, it is preferable to actually follow innovations from the start and score the success factors before the success of the implementation is known.

Conclusion

In this study for the first time ever, an internally validated model was developed that can predict whether innovation implementation projects in radiotherapy will be timely implemented or not. This model enables radiotherapy centres to secure the most critical and manageable success factors before the innovation projects starts. This is doable because only a few statistically significant success factors were relevant for the prediction whether the project will be successful.

This prediction model could be widely applicable to innovate more successfully in radiotherapy.

Following this research, it is recommended to implement the SOP: ‘Preparing for project starts’ into clinical practice, after external validation.