Abstract
Objective
Develop and evaluate a framework for quality improvement which will provide a common approach, terminology and greater likelihood of success in achieving demonstrable and sustained improvement.
Design
Descriptive with mixed methods evaluation.
Setting
Tertiary care academic hospital in Ottawa, Canada.
Participants
Students enrolled in the Quality and Patient Safety Leadership Executive Program at the local university.
Methods
The quality improvement (QI) framework was developed though comparison and coding of key strengths across four commonly applied quality improvement frameworks. Effectiveness of the framework was evaluated through a satisfaction survey of students who were taught the framework, and independent assessment of student improvement initiatives that followed the QI Framework versus initiatives from a prior year who did not.
Main Outcome Measures
Acceptability and effectiveness of the QI Framework.
Results
All participants found the QI Framework to be useful and easy to follow. Independent evaluation of improvement initiatives following the QI Framework, as compared to those that did not, scored higher (95% CI: 3.0 ± 0.49) than the projects completed without the QI Framework (95% CI: 2.2 ± 0.30), P value < 0.01. Scores were higher for cohort 2017/18 in all criteria except scaling and spreading, in which there was no change (2015/16 95% CI: 1.2 ± 0.24, 2017/18 95% CI: 1.2 ± 0.38).
Conclusions
The method we have developed is acceptable and helpful to users, and overall application resulted in higher quality initiatives. We believe this method, which we have branded as The Ottawa Hospital Innovation Framework, can be beneficial in healthcare settings for a variety of change initiatives.
Keywords: quality improvement, methods, program evaluation
Introduction
A quality improvement framework is a stepwise approach to executing quality improvement (QI) projects. A structured framework provides consistency, common thinking and language across organizations. This approach is particularly helpful in healthcare organizations which are complex, adaptive systems, with multiple moving parts and factors influencing care, activities, events and outcomes [1].
There are number of different philosophies, tools and frameworks for health system change. Each of these has pros and cons and varied applicability depending on the healthcare sector and the types of problems it addresses [2]. Among them, we can cite internal continuous improvement models which originate from industry (e.g. Kaizen, Lean, Total Quality Management and Six Sigma) and campaigns, grounded in the theory of large-scale improvement, focusing on predetermined aspects of healthcare in the community. One of the most widely used models is the Model for Improvement methodology [1], which uses rapid cycles of change (Plan-Do-Study-Act [PDSA]) and is the current approach encouraged by the Institute for Healthcare Improvement and others [3, 4].
The literature evaluating QI Frameworks is limited [5, 6]. Most studies focus on the evidence of achieving improved outcomes instead of effectiveness and utility of the framework itself [7]. While outcomes are extremely important, it is also important to understand how best to achieve these outcomes and transfer successful interventions into other settings [8]. Evaluation of these aspects is challenging as a result of the heterogeneity in interventions and contexts [8–10]. There is, however, some evidence that these models lack practical utility and are inconsistently applied in healthcare [6, 7].
Simple approaches, such as the Model for Improvements’ PDSA cycles or Lean, are appealing but insufficient at addressing complexities [6, 7] and have yet to demonstrate improvement through application [11]. Other more complex models, such as Six Sigma, or combinations of other models with Six Sigma, are highly rigorous, and although application of the methodology is observed in very specific instances, there is no evidence to date that this method is a widely applicable overall method for QI within an organization [11]. There are also some niche approaches such as the Comprehensive Unit-Based Safety Program (CUSP), but CUSP is typically not applied beyond surgical specialties, and even among those, other methods are employed to address particular quality problems [12].
The different philosophies, approaches, terminology and tools can result in confusion and, as such, can be a barrier to widespread uptake of QI. To address this concern, we sought to create one simple, effective framework to undertake QI initiatives at our institution. The objective of this paper is to describe our framework as well as how it was developed, disseminated and evaluated. We endeavor to demonstrate that our framework performs well, is acceptable to users and offers key advantages over existing frameworks typically used by healthcare organizations.
Methods
Framework development
We reviewed the most common quality improvement frameworks used at our hospital, namely, the Model for Improvement by the Institute for Healthcare Improvement, Lean, Six Sigma and CUSP by Johns Hopkins Medicine. We reviewed the frameworks in a side-by-side manner to align common components and highlight differences. We coded the key strengths for each framework and grouped these into concepts which became the core components (or steps) of our framework (Fig. 1). This work was also done with careful consideration of the contextual application and varying size and complexity of QI initiatives across our organization.
Figure 1.
The TOH Innovation Framework and alignment to existing methods for improvement.
Once the core components were identified, we enhanced the Framework in two ways: (1) the addition of ‘gates’ and (2) the mapping of key tools. Gates are formal reviews between the project sponsor and the project lead. The criteria at each gate are used to evaluate if and how the project moves forward. The stage-gate approach ensures constant communication and buy-in from stakeholders and sponsors. We identified and aligned the most common, practical tools to each step in the framework to help individuals distinguish tools from the overall framework itself. The tools provided are suggested based on the type of problem being addressed. For example, efficiency and access-related quality issues may be best addressed through use of value stream mapping to identify the root causes. The resulting framework is a step-by-step approach to addressing any type, size or complexity of quality issue encountered. Components of the framework are summarized in Table 1.
Table 1.
Summary of The Ottawa Hospital Innovation Framework components
| Component | Purpose and scope | Strengths |
|---|---|---|
| Define the problem | Documented clarity in the exact quality problem to be addressed; inclusion of the magnitude and importance of the problem; the specific goal to be achieved defined in measurable terms and time-bound (aim statement); scope; process and balancing measures to be applied; project team and executive sponsor support confirmed | Generates clarity and consensus in the specific problem to be addressed; proposes a high-level timeline by respective component of the QI Framework |
| Gate (check-in and approval to proceed to the next step) | Seek approval of the executive sponsor and support of key stakeholders for the problem to be solved by discussing and signing the Project Charter | Garners executive sponsor and key stakeholder support; supports clear and shared understanding of the problem to be addressed |
| Analyze the situation | Identify the main root cause(s) of the problem to be addressed; understanding of the main process/es involving the problem to be solved and the individuals involved in the process/es | Requires root cause(s) to be identified prior to proceeding with any improvement; in this manner improvements are targeted to known cause(s) |
| Gate | Seek approval of the executive sponsor to address the root cause/s proposed (focuses the improvement effort) | Garners executive sponsor agreement and support of the root cause/s to be addressed |
| Test and trial interventions | Address the root cause(s) confirmed through small tests of change (targeted PDSA cycles) to generate evidence of improvement; to trial-controlled change cycles | PDSA cycles are directed and focused on the root cause(s) of the problem reducing unnecessary change cycles and avoiding user change fatigue and confusion |
| Gate | Share results of PDSA cycles; propose and seek approval of the change to made standard work; gather executive support to advance a change to normal operations | Garners executive sponsor agreement to implement a change to operations |
| Lock-in improvements | Implements the change to operations including training, changes to existing (or creates new) policies and procedures; sets in place the (changes to) monitoring systems | Provides consideration for all aspects that are essential to sustainability of the new approach |
| Gate | Provides a dedicated opportunity to check in on the progress made; seek support where change has not been ‘locked-in’; secure support for formal evaluation and spread conversation | Keeps executive sponsorship engagement in the process |
| Evaluate and spread | To evaluate the entire QI initiative from problem identification to current data/information flowing from lock-in monitoring; to set aside time to celebrate successes and reflect on lessons learned; to formally consider possibilities for spread | Allows for dedicated time to evaluate the overall QI initiative from start to finish; enables formal lessons learned reflection exercise; enables dedicated thought to spread and corresponding planning |
Gate = check-in and approval to proceed to the next step.
Framework roll-out
The developed framework was put into use immediately by our hospital Quality Improvement Coordinators who are responsible for supporting quality projects. During the pilot, the Coordinators solicited real-time unstructured feedback from users throughout their projects. In addition, the framework was taught to a small group of hospital leaders as part of an internal leadership program. They applied the framework to their own projects, and again, feedback was sought on the teaching and practical application of the framework. All feedback was used to refine the content and method of delivery.
The revised framework was disseminated to the entire hospital in October 2017, during a hospital-wide information session. The session introduced the framework along with an online platform for all staff to access the framework, tools training and help (among other functions). Monthly ‘Quality & Innovation Showcases’ after the launch highlighted the framework and its components as well as local quality projects that used the framework.
The framework was introduced as part of the curriculum of a Quality and Patient Safety Leadership Executive Program offered through the School of Management at the local university in October 2017. In addition, the framework was introduced as part of the core curriculum of a 1-day course for clinical and administrative staff involved in QI projects as well as a more intensive 4-day course for leaders.
Our experience in applying the framework to a variety of improvement initiatives has garnered support for this framework as a model for innovation, innovation being larger scale change, yet still benefiting from a disciplined methodical approach. For these reasons the framework has been branded as The Ottawa Hospital (TOH) Innovation Framework.
Evaluation
We undertook a two-part evaluation of the acceptability and effectiveness of our framework. Part 1 was a survey of participants who had been taught and applied the new framework. Part 2 was an independent review of quality project that used the new framework compared to projects that did not use the framework.
Setting and Participants
All participants were students enrolled in the Quality and Patient Safety Leadership Executive Program at a local university. This program has been running annually since 2010 and is designed for physician and health service leaders in healthcare organizations who have an interest in improving quality and patient safety. Program participants are sponsored by their home institutions. The program spans 7 months with a total of 63 classroom hours. Each student must commit additional time to undertake a project to improve quality in their organization and prepare a poster to present their initiative and results.
The course content was historically based on The Improvement Guide [13] and did not teach substantial QI methodology. In 2017/18 the new TOH Innovation Framework was incorporated into the program curriculum. Training on the framework was offered in an applied, just-in-time fashion to course participants helping them to learn and then apply the components as they progressed through the execution of their selected QI initiative over the course of the program. As such, the training is a blend of didactic and applied content.
Part 1—Participant survey. We conducted an anonymous survey of all students who were enrolled in the 2017/18 cohort of the Quality and Patient Safety Leadership Executive Program. The survey was disseminated through LimeSurvey [14]. Completion of the survey implied consent. The survey questions were designed to assess the acceptability and effectiveness of the framework (questions in Appendix A).
Part 2—Independent quality project review. Four independent reviewers reviewed and assessed the projects from a year where the QI Framework was not incorporated (2015/16) and the most recent year in which the QI Framework was incorporated (2017/18). The final poster presentation from each project was anonymized and uploaded to a central repository. Standard criteria, described below, were used to assess each project. Reviewers were not blinded to whether the projects were products of the new framework.
Criteria to be used for both 2015/16 and 2017/18 cohort projects were rated on a Likert scale of 1–5, where 1 = totally unclear (or evidence unclear) and 5 = fully clear (or clear evidence). The specific questions are described in Appendix B.
The four reviewers had diverse backgrounds with varying degrees of QI knowledge. One reviewer was a Physician and former Hospital Quality Executive, one was a Research Coordinator, one was a Manager in the Quality Department, and one was a Quality Improvement Coordinator. We attempted to address potential bias by using multiple diverse reviewers and applying the structured review process described.
Analysis
All survey results are presented descriptively. For the independent project review, we calculated mean scores and confidence intervals for each criterion. Mean score across cohorts were compared using t-tests assuming both equal and unequal variance. The non-parametric Wilcoxon signed-rank test was also used to compare medians across cohorts.
Results
Participant survey
We sent the survey to 22 participants and received responses from 14 (64%). Most respondents were physicians (n = 11, 79%) with only a small number of administrators (n = 2, 14%) and 1 nurse (7%). Very few participants had any prior training in QI (n = 3, 21%).
Survey responses (Fig. 2) showed that all respondents (n = 14, 100%) either strongly agreed or agreed that (a) taking the course alleviated anxiety regarding the right way to undertake a QI initiative; (b) the framework provided a simplified stepwise approach to undertaking a QI initiative; (c) the framework was easy to understand and follow; (d) the framework training and the tools taught according to step heightened the participants understanding of concepts, tools and approaches for QI in healthcare; (e) the training enhanced the participants understanding of QI philosophies, frameworks and tools; and (f) the training clarified how philosophies, frameworks and tools can work together for achieving better outcomes to QI initiative efforts. While 67% of respondents (8/12, 2 did not answer) agreed that when ‘compared to previous QI training, the QI Framework, tools and training were superior’, only three participants identified having some previous QI training. Of these, two agreed that the framework was superior compared to their previous training, and one was neutral. The majority of participants (n = 13, 93%) either strongly agreed or agreed that (a) following the framework helped the participant implement a sustainable improvement; (b) following the framework has provided the participant with greater confidence in leading QI initiatives; and (c) following the framework helped the participant communicate the results of my QI initiative.
Figure 2.
Survey responses of the 2017/18 QI Executive Program participants regarding use of the TOH Innovation Framework, n = 14.
Independent assessments of quality improvement projects
Each reviewer independently evaluated 13 projects from the 2015/16 cohort and 13 projects form the 2017/18 cohort. The mean scores for all assessment criterion are shown in Fig. 3. Overall, the projects completed in the 2017/18 cohort (i.e. taught the TOH Innovation Framework) scored higher (95% CI: 3.0 ± 0.49) than the projects completed in cohort 2015/16 (95% CI: 2.2 ± 0.30), P value < 0.01 (two-sample t-tests assuming both equal and non-equal variance). The Wilcoxon signed-rank test also demonstrated significant difference across the medians of the two cohorts (Wilcoxon value = 5, P < 0.01). Mean scores were higher for cohort 2017/18 in all criteria except scaling and spreading in which there was no change (2015/16 95% CI: 1.2 ± 0.24; 2017/18 95% CI: 1.2 ± 0.38). The biggest change (increase in the mean score) was in the presence of a plan for hardwiring (1.4), followed by the extent to which an appreciable change was seen (1.1) and the extent to which the interventions trialed were understood (1.0).
Figure 3.
Independent evaluation of QI initiatives across cohort 1 (2015–2016) not taught TOH Innovation Framework, n = 13, and Cohort 2 (2017–2018) taught TOH Innovation Framework, n = 13.
Results are presented as mean criterion score for all projects from each cohort. 1 = totally unclear (or evidence unclear) and 5 = fully clear (or clear evidence). Overall, evaluators of the projects in the 2017/18 (i.e. those students having been trained on the framework) found that these projects were easier to understand, and one evaluator noted that ‘there were definite improvements in projects in 2017/18 as compared to those from 2015/16’.
Discussion
Healthcare professionals require a simple yet effective method for addressing quality problems. Making improvement happen within the healthcare environment is highly complex, but having a consistent, step-by-step framework to rely upon is an essential component to achieving successful outcomes. Despite several frameworks being available, there are limited studies looking at the effectiveness of QI methods in healthcare. We developed and tested a framework at our institution and performed an evaluation to determine its acceptability and effectiveness. We did so in order to share a common language for QI, to apply rigor and discipline in our improvement efforts and ultimately to be as successful as possible in demonstrating improvements in quality outcomes. We found that in the application described, our framework, being a blend of the best of the methods available globally, coupled with adjustments for the realities of healthcare delivery is an easy to apply and effective method for addressing QI initiatives in the healthcare environment.
The framework we developed calls for disciplined steps in understanding the causes of the quality problem to be solved prior to attempting any change ideas. Change ideas are linked to one or more selected root cause(s) and are then trialed in a controlled way—through application PDSA cycles. As such there are fewer, more targeted PDSAs applied through the framework. The positioning of targeted PDSAs only after a comprehensive understanding of the causes of the overarching problem results in fewer test cycles. Accordingly, Reed and Card’s review of PDSAs argues that a deeper understanding and framing of the problem prior to commencing use of PDSAs is essential [15].
Further, our framework calls for an explicit review at the conclusion of each step with the executive sponsor of the QI initiative in order to share findings, discuss and seek approval and the support necessary to successfully undertake the next step in the framework. These ‘gates’ allow for the stoppage of any QI initiative that is not well positioned to succeed or is unable, for various reasons, to continue. This accountable approach to project continuation is rarely seen in practice where projects often continue that should not.
A strength of our framework is within its development and content; consisting of the best of existing models yet incorporating/or adjusting for the real-life challenges of the healthcare environment. Another strength of the framework is its applicability to quality issues of various sizes and types (e.g. patient safety, effectiveness, efficiency, access or patient-centered related). Our study cohort included different professionals from different institutions, who applied the framework to a variety of projects. This may suggest the framework was applicable across different applications; however our sample size was too small to draw any conclusions. In addition, though our assessment of the new framework evaluated only one improvement effort, it is worth noting that despite not having the benefit of experience with the framework, students were able to apply it with greater project success than the previous year.
Our study is important because we evaluated the framework to gauge its acceptability and effectiveness. We undertook a two-pronged evaluation which included a survey of course participants and a comparison of projects that used and did not use the framework. As with all surveys, there is a risk of non-response bias, which is almost impossible to determine impact. However, we did have a very good response rate and very consistent responses among those who completed the survey. With respect to the comparison between course cohorts, there are also several limitations. Though we performed independent reviews of the completed QI projects, reviewers were not blinded to the framework used, and as such, this may have introduced a bias. Blinding was not possible since the project write-ups for the new framework followed a clear structure. We minimized the impact of possible bias by having multiple independent reviewers and using a structured review process. The cohorts themselves may also have differed; for example, there may have been variation in participant knowledge prior to starting the course. However, students were not granted registration on the basis on knowledge or experience in QI, only a common desire to realize change. It should also be acknowledged that improvements seen in the second cohort may simply have been related to the universal application of a framework and not necessarily our specific framework. Further evaluation of the framework in larger samples and different contexts is required.
Conclusions and recommendations
We demonstrated that the Innovation Framework we developed to guide quality improvement was acceptable to users and performed well at improving quality in healthcare organizations among the participant group that applied it. Of the limited studies to date, there is no clear method for QI that has been evaluated to show enhanced quality outcomes. Rather, current methods such as the PDSA method are argued to be insufficient outside of a broader problem-solving method.
We have formally adopted this framework at our institution. Certainly, in our organization, having one common approach to QI has provided the benefits of common language and enabled healthcare providers across a variety of disciplines to work in a consistent, disciplined manner. Having one common approach will also better enable our organization to continue to perfect our skills in making improvement happen and should ultimately lead to better quality outcomes over time. As next steps, we plan to further evaluate the framework across different disciplines and contexts and hope to evaluate the applicability, utility and overall generalizability of the framework in different institutions.
Acknowledgements
The authors would like to thank Julie Lawson for her assistance in the review and submission of this paper and the Quality Improvement Team at the Ottawa Hospital for their assistance with the management of the project evaluation.
Appendix A—Participant Survey for 2016/17 Telfer QPS Course
Q1. My role is (select one):
a) physician
b) nurse
c) health professional
d) administrator (leader in some health administration capacity)
Q2. I have taken other trainings in QI in healthcare previously (select all that apply).
a) IDEAS—introductory course
b) IDEAS—multiple day, applied course
c) Lean yellow belt
d) Lean green belt
e) Other (i.e. value stream mapping, root cause analysis, etc.)—list (free text)
Q3. Compared to previous training, I found the QI Framework and tools taught according to step to be superior.
a) Strongly agree
b) Agree
c) Neutral
d) Disagree
e) Strongly disagree
Q4. Taking the course has alleviated my anxiety regarding the right way to undertake a QI initiative.
a) Strongly agree
b) Agree
c) Neutral
d) Disagree
e) Strongly disagree
Q5. The QI Framework provided a simplified stepwise approach to undertaking a QI initiative.
a) Strongly agree
b) Agree
c) Neutral
d) Disagree
e) Strongly disagree
Q6. Following the QI Framework helped me implement a sustainable improvement.
a) Strongly agree
b) Agree
c) Neutral
d) Disagree
e) Strongly disagree
Q7. The QI Framework is easy to understand and follow.
a) Strongly agree
b) Agree
c) Neutral
d) Disagree
e) Strongly disagree
Q8. The QI Framework training and the tools taught according to step heightened my understanding of concepts, tools and approaches for QI in healthcare.
a) Strongly agree
b) Agree
c) Neutral
d) Disagree
e) Strongly disagree
Q9. The training enhanced my understanding of QI philosophies, frameworks and tools.
a) Strongly agree
b) Agree
c) Neutral
d) Disagree
e) Strongly disagree
Q10. The training enhanced my understanding of QI philosophies, frameworks and tools.
a) Strongly agree
b) Agree
c) Neutral
d) Disagree
e) Strongly disagree
Q11. The training clarified how philosophies, frameworks and tools can work together for achieving better outcomes to QI initiative efforts.
a) Strongly agree
b) Agree
c) Neutral
d) Disagree
e) Strongly disagree
Q12. Following the QI Framework has provided me with greater confidence in leading QI initiatives.
a) Strongly agree
b) Agree
c) Neutral
d) Disagree
e) Strongly disagree
Q13. Following the QI Framework helped me communicate the results of my QI initiative.
a) Strongly agree
b) Agree
c) Neutral
d) Disagree
e) Strongly disagree
Appendix B—Criteria to assess projects
Questions*:
Rate the extent to which you understood the problem they were trying to solve.
Rate the extent to which you clearly understand interventions trialed.
What evidence was there that the findings or the lessons learned led to iterative improvement cycles, i.e. adopt, abandon, adapt?
Rate the extent to which you see appreciable changes as a result [of the work].
Was there a plan for hardwiring?
Was there a plan for scaling and spreading?
*Rated on a Likert scale of 1–5, where 1 = totally unclear (or evidence unclear) and 5 = fully clear (or clear evidence).
Contributor Information
Samantha Hamilton, Innovation & Quality, The Ottawa Hospital, Civic Campus, 1053 Carling Ave Ottawa, ON, K1Y 4E9.
Alison Jennings, Clinical Epidemiology Program, Ottawa Hospital Research Institute, 1053 Carling Ave., Ottawa, ON, K1Y 4E9.
Alan J Forster, Innovation & Quality, The Ottawa Hospital, Civic Campus, 1053 Carling Ave Ottawa, ON, K1Y 4E9; Clinical Epidemiology Program, Ottawa Hospital Research Institute, 1053 Carling Ave., Ottawa, ON, K1Y 4E9.
References
- 1. Braithwaite J. Changing how we think about healthcare improvement. BMJ 2018;361:k2014. doi: 10.1136/bmj.k2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Øvretveit J. What Are the Advantages and Limitations of Different Quality and Safety Tools for Health Care? 2005. http://www.euro.who.int/document/e87577.pdf (20 March 2019, last date accessed). [DOI] [PubMed]
- 3. Health Quality Ontario Quality Improvement Science. 2013. www.hqontario.ca (30 January 2019 last date accessed).
- 4. Patient Safety Institute Improvement Frameworks: Getting Started Kit. 2015. www.saferhealthcarenow.ca (30 January 2019 last date accessed).
- 5. Spencer E, Walshe K. National quality improvement policies and strategies in European healthcare systems. Qual Saf Health Care 2009;18:i22–7. doi: 10.1136/qshc.2008.029355. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Kraft S, Carayon P, Weiss J et al. A simple framework for complex system improvement. Am J Med Qual 2015;30:223–31. doi: 10.1177/1062860614530184. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Taylor MJ, McNicholas C, Nicolay C et al. Systematic review of the application of the plan-do-study-act method to improve quality in healthcare. BMJ Qual Saf 2014;23:290–8. doi: 10.1136/bmjqs-2013-001862. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Øvretveit J, Gustafson D. Evaluation of quality improvement programmes. Qual Saf Health Care 2002;11:270–5. doi: 10.1136/QHC.11.3.270. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Walshe K, Freeman T. Effectiveness of quality improvement: learning from evaluations. Qual Saf Health Care 2002;11:85–7. doi: 10.1136/QHC.11.1.85. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Busetto L, Luijkx K, Vrijhoef HJM. Development of the COMIC model for the comprehensive evaluation of integrated care interventions. Int J Care Coord 2016;19:47–58. doi: 10.1177/2053434516661700. [DOI] [Google Scholar]
- 11. Deblois S, Lepanto L. Lean and six sigma in acute care: a systematic review of reviews. Int J Health Care Qual Assur 2016;29:192–208. doi: 10.1108/IJHCQA-05-2014-0058. [DOI] [PubMed] [Google Scholar]
- 12. Brown R, Grehan P, Brennan M et al. Using lean six sigma to improve rates of day of surgery admission in a national thoracic surgery department. International J Qual Health Care 2019;31(Supplement_1):14–21. doi: 10.1093/intqhc/mzz083. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Langley GL, Moen R, Nolan KM et al. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance (2nd Edition). San Francisco, California, USA: Jossey-Bass Publishers; 2009.
- 14. Limesurvey GmbH LimeSurvey: An Open Source survey tool. http://www.limesurvey.org. https://community.limesurvey.org/licence-trademark/#citation.
- 15. Reed JE, Card AJ. The problem with plan-do-study-act cycles. BMJ Qual Saf 2016;25:147–52. doi: 10.1136/bmjqs-2015-005076. [DOI] [PMC free article] [PubMed] [Google Scholar]