Table 3.

Qualitative analysis of discussion data for each feature

Feature	Panels discussing this feature (no. of comments)	Summary of main themes and comments
1. Systematic data guided activities	A and C (10)	• Should this criterion be distinguishing CQI from research? • Is the use of aims and measures an example of a data-guided activity? Illustrative Comment: ‘Decisions should be data-guided, but aims and measures should be established in a manner that facilitates identification, selection, collection and use of relevant data.’
2. Aiming to change routine work	B, C, and D (8)	• Is CQI changing routine practices or introducing new processes? Is there a difference between the two? Illustrative Comment: ‘I thought that this might exclude QI efforts that aim to change processes that are not routine/daily (e.g. increasing seasonal flu shots…time limited project) but this would hinge on how you interpret ‘routine/daily’.
3. Creating a culture of quality improvement	A, B, and D (16)	• While culture is often critical to the success and sustainability of CQI, it may not be important to its definition. • Culture change may not be the primary purpose of a QI intervention, although it may be necessary for success. Illustrative Comment: ‘Some CQI projects can be initiated and completed without culture change. However, developing an organization that incorporates CQI in all of its work requires culture change.’
4. Specific, pre-defined aims	B and D (11)	• Does having a pre-defined aim imply that issues identified during the implementation will be ignored? • Does ‘pre-defined’ indicate predetermined by others (versus those making the improvements)? • Does this feature lead to incremental versus fundamental or visionary change? Illustrative Comment: ‘If you don't have an identified aim, how can you plan the improvement or know if it was successful? ….However, if ‘pre-identified’ means identified by somewhat [stet] in a QI office or at the top of the hierarchy in the organization…I completely agree that this is not an important part of CQI.’
5. Using evidence relevant to the problem	A and C (10)	• While CQI projects that are not based on strong evidence may have less chance of success, improvement may need to happen in the absence of strong evidence. • CQI can be used to generate evidence that an intervention is effective. • In the absence of existing evidence, expert judgment may be used to evaluate the potential of a new initiative. Illustrative Comment: ‘The best evidence for the change is whether it is effective in the current context. Prior evidence, if available, should be consulted, but (a) it is not always available and (b) even if available is not always relevant.’
6. Designing with local conditions in mind	B and D (7)	• What are the implications when an intervention is developed elsewhere and implemented locally using CQI? • Do senior management roll-outs of changes across clinics qualify as CQI? Illustrative Comment: ‘In collaborative QI projects involving organization from multiple organizations [stet] the design and description of changes is generic and doesn't address local conditions. Then each participating organization tests and adapts the changes to their local conditions.’
7. Iterative development and testing	A, B, and D (11)	• Should implementation of improvements without iterative testing be considered CQI? • If the improvement requires only one cycle of testing, can it be considered CQI? • If an improvement is spread and embedded in practice without iterative testing, can it be considered CQI? Illustrative Comment: ‘QI is not equivalent to CQI. It seems to me that one defining feature of CQI to distinguish it from other types of QI activities is the iterative cycle….Implementing someone else's intervention may be important, may be laudable, and may represent a significant challenge, but if it is not done iteratively in some manner of PDCA [Plan-Do-Check-Act] then is it CQI? My vote is no.’
8. Multidisciplinary teams from target organizations	C and D (20)	• Can CQI sometimes be carried out by just one discipline? • Does the scale and scope of the project determine the need for multiple disciplines? • Is involvement of stakeholders equivalent to multidisciplinary teams? Illustrative Comment: ‘It is hard to think of a process in health care or public health that is not multi-disciplinary and/or cross functional. Referring to multi-disciplinary is a generic feature of improvement (the exceptions are far less frequent than the rule).’
9. Data feedback to implementers	C (6)	• Is continuous feedback an important aspect of the ethical conduct of research? • Is it important to report in publications? • Not feeding back information to those that are part of the QI process may be exploitation by researchers. Illustrative Comment: ‘…this seemed self-evident as part of the research, but I struggled to put it in the context of the published paper. I would still rate a 5 to have a description of feedback included because that too, models the process and can act as dissemination of good practice.
10. Specific named improvement methods	A and C (12)	• If an intervention references a specific method it is easier to label as CQI for literature review. • The specific method (e.g. Lean, Six Sigma) varies in how it is used at each site, and projects may combine methods from different models, so the name is not helpful as a defining feature of CQI. Illustrative Comment: ‘I think the challenge is how to report methods in a more standardized way.’
11. Set of specific changes	C (10)	• Does ‘set of specific changes’ imply pre-determined changes? If so, not appropriate as CQI. • Is ‘sustaining the gains’ an outcome, or a CQI requirement? Illustrative Comment: ‘ … I interpreted this a [stet] merely the importance of embedding improvements into routine work processes. If predetermined were part of the feature then I would definitely rate this extremely low as that would very counterproductive.’