Abstract
Background:
In 2013, the Canadian Institutes of Health Research funded 12 community-based primary health care research teams to develop evidence-based innovations. We aimed to explore the scalability of these innovations.
Methods:
In this cross-sectional study, we invited the 12 teams to rate their evidence-based innovations for scalability. Based on a systematic review, we developed a self-administered questionnaire with 16 scalability assessment criteria grouped into 5 dimensions (theory, impact, coverage, setting and cost). Teams completed a questionnaire for each of their innovations. We analyzed the data using simple frequency counts and hierarchical cluster analysis. We calculated the mean number and standard deviation (SD) of innovations that met criteria within each dimension that included more than 1 criterion. The analysis unit was the innovation.
Results:
The 11 responding teams evaluated 33 evidence-based innovations (median 3, range 1–8 per team). The innovations focused on access to care and chronic disease prevention and management, and varied from health interventions to methodological innovations. Most of the innovations were health interventions (n = 21), followed by analytical methods (n = 4), conceptual frameworks (n = 4), measures (n = 3) and strategies to build research capacity (n = 1). Most (29) met criteria in the theory dimension, followed by impact (mean 22.3 [SD 5.6] innovations per dimension), setting (mean 21.7 [SD 8.5]), cost (mean 17.5 [SD 2.1]) and coverage (mean 14.0 [SD 4.1]). On average, the innovations met 10 of the 16 criteria. Adoption was the least assessed criterion (n = 9). Most (20) of the innovations were highly ranked for scalability.
Interpretation:
Scalability varied among innovations, which suggests that readiness for scale up was suboptimal for some innovations. Coverage remained largely unaddressed; further investigation of this critical dimension is necessary.
The development and implementation of evidence-based innovations have been strongly incentivized as part of primary care reform in Canada.1 The research focus has been on generating enough evidence of sufficient rigour to qualify innovations as evidence-based.2–4 However, there is a growing gap between development of evidence-based innovations as research projects and their widespread implementation as standard care.5–7 Scale up or spread is a form of knowledge translation that aims to increase the reach and adoption of innovations.8–13 Successful scale up follows several steps: 1) assessment of scalability (potential for scale up) of the innovation (e.g., effectiveness, cost-effectiveness), 2) development of a scale-up strategy, 3) strategy implementation, 4) strategy evaluation and 5) assuring sustainability.13–16 Rigorous evaluation of scale up includes measuring coverage (number of people, organizations or systems that adopt the innovation over the number targeted).2,17 In Canada, however, there are few systematic efforts to facilitate and support scale up and few theoretical, conceptual or practical frameworks for guiding the first step, scalability assessment.13,14 This leads to a lag between evidence generation and scale up, and denies patients and communities access to effective services.7,18–20
In 2013, 12 research teams were funded by the Canadian Institutes of Health Research to conduct cross-jurisdictional research in improving access to community-based primary health care for vulnerable populations, and chronic disease prevention and management.21 As a funding condition, teams were required to collaborate in sharing their findings and lessons learned, build capacity and plan for scale up.22 In the present study, we aimed to explore the scalability of the teams’ evidence-based innovations.
Methods
Study design
In this cross-sectional study, we invited the 12 research teams to rate the scalability of their evidence-based innovations. An evidence-based innovation was defined as a program, model, approach, tool, instrument, indicator, algorithm, service, idea, policy or practice whose evidence base has been established.23,24
Setting and participants
We conducted the study between August and December 2017 together with the teams. The teams covered all of Canada except for 1 province (Saskatchewan) and 1 territory (Yukon Territory).21,22 They included over 100 stakeholders in community-based primary health care.21,22 Their research was programmatic, i.e., each team worked on varied but conceptually linked projects bound together by the themes of access to care for vulnerable populations, and chronic disease prevention and management.21,22
Data collection
We created a 1-page self-administered questionnaire containing the key elements for assessing the scalability of the teams’ evidence-based innovations (Appendix 1, available at www.cmajopen.ca/content/6/4/E520/suppl/DC1). We designed the questionnaire based on the results of a systematic review of scale-up strategies in primary care2 and the recommendations of 2 scale-up guides.13,14 The teams were invited to rate the scalability of their evidence-based innovations and were asked for details about their team and the name, type and aim of the innovation. The questionnaire included 16 criteria for assessing scalability, grouped into 5 dimensions: 1) the theory used to develop the innovation, 2) impact (e.g., has efficacy been assessed), 3) likely coverage of the innovation (e.g., has adoption been assessed), 4) alignment of the innovation with the setting (e.g., is it compatible with similar interventions in the same setting) and 5) cost (e.g., is scale up affordable). For each criterion, there were 5 response options: 1) “Yes” (criterion was assessed), 2) “No” (criterion was not assessed), 3) “UE” (criterion was under evaluation), 4) “NP” (criterion assessment was not planned) and 5) “NA” (not applicable). Space was provided for additional comments on each criterion. The questionnaire was piloted by 1 of the 12 teams (S.T.W.’s team) and was revised to improve understanding of the instructions and examples of evidence-based innovations. All teams were asked to return their completed responses by email. A maximum of 3 reminder messages were sent no later than 3 weeks after the first message. A team could rate more than 1 of their innovations (1 questionnaire each).
Data analysis
Two of the authors (A.B.C. and H.T.V.Z.) independently classified the innovations into 1 of 5 mutually exclusive types according to predefined themes. Any discrepancies were resolved through consensus with the senior author (F.L.). The themes corresponded to the following types of evidence-based innovations:
Health intervention: an act performed for, with or on behalf of a person or a population to improve, maintain, promote or modify health functioning or health conditions (e.g., preventive strategy, screening program, decision aid)25
Analytical method: a generic or systematic process combining the scientific method with the use of a formal process to solve a research problem26,27
Conceptual framework: a set of concepts or abstractions linked and arranged rationally according to their relevance to a common theme28
Measure: any instrument, measure or indicator29
Strategy to build research capacity: a process of individual or institutional development that leads to a greater ability to perform useful research.30,31
Team respondents were involved in all steps of the study. Together, we grouped the response options for the 16 scalability assessment criteria into 3 categories: criterion met (responses corresponding to “criterion was assessed” and “criterion was under evaluation”), criterion not met (responses corresponding to “criterion was not assessed” and “criterion assessment was not planned”) and not applicable. Thus, each innovation was scored on the number of criteria met. Then we ranked the 16 criteria according to the number of innovations that met each of these criteria. For the dimensions that include more than 1 criterion, we calculated the mean number and standard deviation (SD) of innovations per dimension for which criteria were met. We used analysis of variance to compare the number of criteria met per innovation.
We analyzed the data using simple frequency counts and hierarchical cluster analysis. The unit of analysis was the innovation. As we had very little missing data, no specific missing-case analyses were conducted. We analyzed the data using SAS version 9.4 (SAS Institute). We conducted a hierarchical cluster analysis among the innovations using the SAS CLUSTER procedure.32 This allowed us to group innovations into the most homogeneous clusters possible, based on the number of criteria met. Thus, the hierarchical clustering analysis was aimed to create clusters of innovations but not to analyze existing clusters. The objective was to rank innovations in order of their scalability. We used the average linkage method, whereby the distance between 2 clusters is the average of the distances of all pairs of innovations, 1 in each cluster. 32,33 We estimated the optimal number of clusters using the pseudo F and pseudo t2 statistics.32
Ethics approval
Our study assessed research innovations and did not collect data for human participants; thus, ethics review was not needed.34
Results
Participants
Eleven of the 12 teams reported at least 1 evidence-based innovation (Figure 1). One team did not respond owing to lack of time and capacity. We received information for 33 innovations (median 3, range 1–8 per team). As shown in Table 1, 23 innovations focused on chronic diseases and/or access to resources (e.g., health care services, social services and complex care needs).
Table 1:
Type of innovation | Scalability ranking* | No. of scalability assessment criteria met† | Reported aim(s) of innovation |
---|---|---|---|
Health intervention | High | 16 | Increase access to primary health care for people/populations that are underserved by, and struggle to connect with, traditional primary health care services; improve coordination and collaboration of primary health care service providers when providing care |
Health intervention | High | 14 | Multifaceted interprofessional community-based self-management program for older adults with diabetes and multiple chronic conditions |
Health intervention | High | 14 | Introduce innovative team work for the management of patients with multimorbidity |
Health intervention | High | 14 | Enhance the new patient’s capacity to access the clinic and understand how it functions |
Health intervention | High | 13 | Improve diabetes prevention and management by leveraging existing resources |
Health intervention | High | 13 | Interprofessional community-based self-management program for older adults with vascular conditions and multiple chronic conditions |
Health intervention | High | 13 | Actionable information on primary care for key stakeholders; sustainable performance portrait (through ongoing funding); innovation through information to stakeholders |
Health intervention | High | 13 | Reduce the impact of vascular disease on the health of Canadians |
Health intervention | High | 12 | Integrate chronic disease prevention and management by a team in primary care practice |
Health intervention | High | 12 | Develop knowledge, skills and abilities to work with primary care practices to support people experiencing social barriers in reaching community health and social resources |
Health intervention | High | 11 | Identify patients at increased risk for breast and colorectal cancer and personalize screening and management; help patients with breast cancer make treatment decisions |
Health intervention | High | 11 | Integrated approach for chronic diseases |
Health intervention | High | 11 | Improve communication between primary care providers and cancer specialists; improve continuity of care among patients with cancer |
Health intervention | High | 11 | Type of patient referrals, navigation services provided, community health and social resource needs |
Health intervention | High | 11 | Provide rural remote primary care |
Health intervention | High | 10 | Toolkit to support caregivers of older adults with dementia and multiple chronic conditions with changes they are experiencing |
Health intervention | Medium | 8 | Ensure meaningful engagement and participation of people with lived experience in the research process, and ensure that people with lived experience are adequately compensated for their time and energy devoted to the research |
Health intervention | Medium | 5 | Explore how people might use publicly reported primary care information through a series of citizen-patient deliberations in 3 Canadian provinces |
Health intervention | Medium | 4 | Lead to awareness among allied health care professionals and policy shifts in health care systems regarding mental health care for youth and support for caregivers |
Health intervention | Low | 1 | Present the clinical cancer system and experiences of patients with breast and colorectal cancer in Canada through synthesis maps to be used as knowledge-translation tools |
Health intervention | Low | 0 | Profile Canadian initiatives aimed at improving continuity of care between primary care providers and cancer specialists |
Analytical method | High | 12 | Piloted a practice-based automated patient survey system by telephone or email with consenting patients |
Analytical method | Medium | 9 | Comparability of health administrative data indicator definitions across provinces |
Analytical method | Medium | 6 | Gather narratives/stories of youth mental health journeys from multivocal perspectives |
Analytical method | Medium | 5 | Use analytical modelling to support children and youth with mental health conditions |
Conceptual framework | High | 13 | Enable health care providers to implement integrated care for people with complex health needs |
Conceptual framework | High | 10 | Inform implementation of patient-centred interventions to improve care of patients with multimorbidity |
Conceptual framework | Medium | 8 | Support innovation and transformation in First Nations health |
Conceptual framework | Medium | 6 | Understand mental health from a First Nations perspective |
Measure | Medium | 9 | Identify people with multimorbidity in primary care |
Measure | Medium | 8 | Assess the integration, coordination and transitions of care |
Measure | Medium | 6 | Assess the correlations of sex and gender with patient outcomes |
Research capacity | High | 10 | Build research capacity |
Scalability ranking groups are based on hierarchical cluster analysis.
Range 0–16.
Types of evidence-based innovations
The majority (21) of the evidence-based innovations were health interventions (e.g., a community partnership program aiming to reduce the impact of vascular disease on the health of Canadians35), followed by analytical methods (4 innovations) (e.g., piloting an automated practice-based patient survey system by telephone or email), conceptual frameworks (4 innovations) (e.g., a guide to implementing integrated care), measures (3 innovations) (e.g., a 21-item measure for identifying people with multimorbidity in primary care) and strategies to build research capacity (1 innovation).
Scalability assessment dimensions
Use of a theory to develop the innovation (1 criterion) was the dimension in which scalability assessment criteria were most often met (n = 29) (e.g., the Consolidated Framework for Implementation Research), followed by impact (mean 22.3 [SD 5.6] innovations per dimension), setting (mean 21.7 [SD 8.5]), cost (mean 17.5 [SD 2.1]) and coverage (mean 14.0 [SD 4.1]) (Table 2). Within the coverage dimension, the criteria of reach, adoption and maintenance were frequently reported as not applicable (n = 12, 12 and 14, respectively) (Supplementary Table A1, Appendix 1), with the most commonly reported reason being that the innovation was “not an intervention.”
Table 2:
Assessment dimension/criteria | No. (%) of innovations that met criteria | Mean* no. of innovations that met criteria per dimension ± SD |
---|---|---|
Use of theory: Evidence-based innovation developed with theory | 29 (88) | NA |
Impact | 22.3 ± 5.6 | |
Acceptability | 22 (67) | |
Feasibility | 26 (79) | |
Adaptability | 19 (58) | |
Efficacy | 13 (39) | |
Effectiveness | 27 (82) | |
Results documented | 27 (82) | |
Setting | 21.7 ± 8.5 | |
Implemented in setting comparable to target setting | 28 (85) | |
Compatibility with similar evidence-based innovations in target settings | 12 (36) | |
Consistency with policy directives | 25 (76) | |
Cost | 17.5 ± 2.1 | |
Cost-effectiveness | 16 (48) | |
Resources needed for scale up (affordability) | 19 (58) | |
Coverage | 14.0 ± 4.1 | |
Reach (numerator and denominator) | 14 (42) | |
Adoption (numerator and denominator) | 9 (27) | |
Fidelity | 19 (58) | |
Maintenance | 14 (42) |
Note: NA = not applicable, SD = standard deviation.
Sum of the number of innovations that met the criteria in each dimension divided by the total number of criteria in that dimension.
On average, 10 (SD 4) of the 16 scalability assessment criteria were met by the 33 innovations. Adoption was the least assessed criterion (9 innovations). The number of criteria that were met in the coverage dimension varied with the type of innovation (p = 0.005), with health interventions likely to meet the most criteria.
Ranking for scalability
Using hierarchical cluster analysis, we classified the 33 innovations into 3 groups (pseudo F = 73.5, pseudo t2 = 8.0): those whose scalability was ranked as high (n = 20), those whose scalability was ranked as medium (n = 11) and those whose scalability was ranked as low (n = 2) (Table 1). The mean number of scalability assessment criteria met for the 3 groups was 12 (SD 2), 7 (SD 2) and 1 (SD 1), respectively. A high ranking indicated that the team had collected diverse important information relevant to making a decision about the scale up of the innovation. Nine of the 11 responding teams ranked at least 1 of their innovations as having high potential for scale up (median 1, range 0−5 per team). Most (16) of the 20 high-ranking innovations were health interventions.
Interpretation
In this study, we explored the scalability of a large sample of evidence-based innovations in Canadian community-based primary health care. The majority of innovations were health interventions, and most ranked high for scalability. However, few of the innovations met the scalability assessment criteria relating to coverage, a dimension essential for scalability. These findings lead us to make the following observations.
First, the innovations that ranked highest for scalability were health interventions. For example, the pop-up health and community service event met all of the scalability assessment criteria.36 As the innovation had already been implemented and assessed in several different settings (e.g., a seniors’ centre, the Indigenous Early Intervention Organization, an elementary school), the team was likely to have collected a variety of relevant data. A crucial element of successful scale up is user-informed assessment of the environment to determine whether there is a fit between the target units and the innovation.16,37 A danger in health care system reform is the rolling-out of innovations as population-wide “one-size-fits-all” solutions. This project had clearly based its scaled-up implementation on individual and community-based needs.1,36
Second, innovations that were not health interventions were overall ranked lower in scalability. The diversity of innovations developed by the teams (from health interventions to methodological innovations) reflects the inherent complexity of community-based primary health care research, not all of which can be appropriately or easily scaled up. Teams whose innovations consisted of a conceptual framework or an analytical method, for example, were less concerned about widespread implementation of their innovations as they were not necessarily designed to directly improve health functioning or alleviate conditions.38 In addition, the diffusion of methodological innovation often takes a more complex, intuitive and organic path that is more horizontal than top-down. 8,39,40 However, there is value in assessing, adapting and scaling up methodological innovations for wider use by researchers,39,41–44 and there are too few opportunities for researchers to develop and experiment with methods for addressing social and clinical research questions.39,45 The usefulness of all scaled-up innovations will ultimately depend on the strength of the methodology developed.
Third, we found that coverage was the dimension in which scalability assessment criteria were least likely to be met, and adoption was the least assessed criterion. This may be in part because some innovations were not yet fully implemented or were designed to remain local rather than scalable. Lack of scalability frameworks to support research design and lack of sufficient time and resources could be other reasons. Moreover, some of the concepts relating to the science of scale up (e.g., adoption, adaptation, maintenance, spread and scale up itself) are as yet ill-defined and undertheorized in knowledge translation.46 Some of these concepts were unfamiliar to some of the stakeholders responding to our questionnaire. This highlights the need to refine and standardize terminology relating to scale-up science.2,47
Finally, it is difficult to compare our results with the literature, as we found few studies that analyzed primary health care innovations using scalability assessment criteria. The authors of a study assessing scaled-up public health interventions found that 25% of interventions had not been trialled, and, in 5%, efficacy had not been assessed, but they did not address coverage.48 The authors of a systematic review on scaling up evidence-based innovations in primary care noted that coverage was least likely to be assessed.2 Coverage of targeted users or settings is at the heart of scalability.2,13,17 For innovations to have a substantial impact, they need to be adopted by a large enough population over a sustained period.13 Developers of innovations can evaluate their programs using the Reach, Effectiveness, Adoption, Implementation, and Maintenance Qualitative Evaluation for Systematic Translation (RE-AIM QuEST) framework to specifically explore the coverage potential of their innovations.49,50
Limitations
First, in the interests of brevity, our questionnaire did not adequately explain every dimension of assessing scalability. Two teams contacted us for further explanation. Although the questionnaire was not validated, it was created through a group process (with 3 teams) and was pretested with at least 3 innovations. Second, we collected data on the existence of assessment criteria but not on the results of those assessments. Thus, based on our data, we cannot confidently label the innovations as scalable or not. However, our data clearly show that some teams collected important information that would facilitate scale up of their innovations. Moreover, feedback from the team members indicated that they found the experience constructive and informative for the purposes of future scale-up planning. Third, the questionnaires were completed by team stakeholders who, although best qualified to answer detailed questions about their projects, were also researchers in the study. This may have reduced objectivity. Also, all teams had end-user partners, and if these partners had completed questionnaires as well, they would have added an important perspective on the project’s scalability. Finally, scores obtained with the hierarchical cluster analysis are specific to the innovations used, which does not allow for generalization of results. As a first exercise to rank the scalability of evidence-based innovations in community-based primary health care in Canada, these findings will enable us to refine the language of scaling up, adapt the scalability assessment criteria, improve our questionnaire and validate it for future studies.
Conclusion
We explored scalability assessment criteria among a large sample of evidence-based innovations in Canadian community-based primary health care. Overall, teams had important information about their innovations that they could use to plan scale up. However, coverage is a critical dimension that remains largely unaddressed. Our findings provide concrete data on evaluating innovations for larger-scale use and contribute important new understandings of scalability assessment with relevance to a broad group of stakeholders. The questionnaire itself served as a knowledge-translation tool, motivating teams to talk with their research communities and decision-makers about scalability. Finally, our findings inform stakeholders about how and why scalability should be evaluated and reported as an outcome measure in project design and grant submission.
Supplementary Material
Acknowledgements
The authors thank the following team-nominated principal investigators: Marshall Godwin, Eva Grunfeld, Jeannie Haggerty, Stewart Harris, Janusz Kaczorowski, Moira Stewart and Walter Wodchis. They also thank the members of the teams for their contribution: ACCESS-MH (Atlantic Canada Children’s Effective Service Strategies in Mental Health), ACHRU (Aging, Community and Health Research Unit), CanIMPACT (Canadian Team to Improve Community-Based Cancer Care along the Continuum), C-ChAMP (Canadian Chronic Disease Awareness and Management Program), CircHSIT (Circumpolar Health System Innovation Team), FORGE AHEAD (Transformation of Indigenous Primary Healthcare Delivery), iCOACH (Implementing Integrated Care for Older Adults with Complex Health Needs), IMPACT (Innovative Models Promoting Access-to-Care Transformation), iPHIT (Innovation in Community-Based Primary Healthcare Supporting Transformation in the Health of First Nations and Rural/Remote Manitoba Communities), LHIV (Living with HIV Innovation Team), PACE in MM (Patient-Centred Innovations for Persons with Multimorbidity) and TRANSFORMATION (Improving the Science and Reporting of Performance in Primary Care). Finally, the authors thank Louisa Blair, English-language editor, for her kind help with the manuscript.
Footnotes
Competing interests: None declared.
This article has been peer reviewed.
Contributors: Ali Ben Charif, Kasra Hassani, Sabrina Wong, Hervé Zomahoun, Martin Fortin, Adriana Freitas, Kathryn Nicholson, Jenny Ploeg and France Légaré contributed to the study conception and design. Kasra Hassani and Sabrina Wong coordinated data collection. The 12 research teams contributed to data collection. Ali Ben Charif analyzed the data and drafted the manuscript, overseen by France Légaré. All of the authors contributed to the interpretation of data, revised the manuscript critically for important intellectual content, gave final approval of the version to be published and agreed to be accountable for all aspects of the work.
Funding: This study was funded by Canadian Institutes of Health Research grants awarded to the 12 research teams. Ali Ben Charif was funded by the Fonds de recherche du Québec − Santé. Ali Ben Charif and Hervé Zomahoun were funded by the Quebec Strategy for Patient-Oriented Research (SPOR) Support for People and Patient-Oriented Research and Trials (SUPPORT) unit. France Légaré holds a Tier 1 Canada Research Chair in Shared Decision Making and Knowledge Translation, and Adriana Freitas was funded by this chair.
Disclaimer: Only the authors have responsibility for the information provided or views expressed in this article.
Supplemental information: For reviewer comments and the original submission of this manuscript, please see www.cmajopen.ca/content/6/4/E520/suppl/DC1.
References
- 1.Fooks C. Implementing primary care reform in Canada: barriers and facilitators. Montréal: McGill-Queen’s University Press; 2003. [accessed 2018 Apr. 26]. Available: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.502.6522&rep=rep1&type=pdf. [Google Scholar]
- 2.Ben Charif A, Zomahoun HTV, LeBlanc A, et al. Effective strategies for scaling up evidence-based practices in primary care: a systematic review. Implement Sci. 2017;12:139. doi: 10.1186/s13012-017-0672-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Fixsen D, Blase K, Metz A, et al. Statewide implementation of evidence-based programs. Except Child. 2013;79:213–30. [Google Scholar]
- 4.Milat AJ, Bauman A, Redman S. Narrative review of models and success factors for scaling up public health interventions. Implement Sci. 2015;10:113. doi: 10.1186/s13012-015-0301-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Haynes B, Haines A. Barriers and bridges to evidence based clinical practice. BMJ. 1998;317:273–6. doi: 10.1136/bmj.317.7153.273. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Kristensen N, Nymann C, Konradsen H. Implementing research results in clinical practice — the experiences of healthcare professionals. BMC Health Serv Res. 2016;16:48. doi: 10.1186/s12913-016-1292-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Bégin M, Eggertson L, Macdonald N. A country of perpetual pilot projects. CMAJ. 2009;180:1185, E88–9. doi: 10.1503/cmaj.090808. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Shaw J, Tepper J, Martin D. From pilot project to system solution: innovation, spread and scale for health system leaders. BMJ Leader. 2018;2:87–90. [Google Scholar]
- 9.Massoud MR, Donohue KL, McCannon CJ. Options for large-scale spread of simple high-impact interventions [technical report] Chevy Chase (MD): USAID Health Care Improvement Project; Chevy Chase (MD): University Research Co, LLC; Washington (DC): United States Agency for International Development; Geneva: World Health Organization; Boston: Harvard School of Public Health; 2010. [accessed 2018 Apr. 26]. Available: https://www.urc-chs.com/resources/options-large-scale-spread-simple-high-impact-interventions. [Google Scholar]
- 10.Eaton J, McCay L, Semrau M, et al. Scale up of services for mental health in low-income and middle-income countries. Lancet. 2011;378:1592–603. doi: 10.1016/S0140-6736(11)60891-X. [DOI] [PubMed] [Google Scholar]
- 11.How-to guide: sustainability and spread. Boston: Institute for Healthcare Improvement; 2005. [accessed 2018 Apr. 26]. Available: www.ihi.org/resources/pages/tools/howtoguidesustainabilityspread.aspx. [Google Scholar]
- 12.Albury D, Beresford T, Dew S, et al. Against the odds: successfully scaling innovation in the NHS. London (UK): Innovation Unit and The Health Foundation; 2018. [accessed 2018 Feb 8]. Available: http://kingsfund.blogs.com/health_management/2018/01/against-the-odds-successfully-scaling-innovation-in-the-nhs.html. [Google Scholar]
- 13.Milat AJ, Newson R, King L, et al. A guide to scaling up population health interventions. Public Health Res Pract. 2016;26:e2611604. doi: 10.17061/phrp2611604. [DOI] [PubMed] [Google Scholar]
- 14.Nine steps for developing a scaling-up strategy. Geneva: World Health Organization; 2010. [accessed 2018 Apr. 26]. Available: www.who.int/reproductivehealth/publications/strategic_approach/9789241500319/en/ [Google Scholar]
- 15.Barker PM, Reid A, Schall MW. A framework for scaling up health interventions: lessons from large-scale improvement initiatives in Africa. Implement Sci. 2016;11:12. doi: 10.1186/s13012-016-0374-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Bradley EH, Curry LA, Taylor LA, et al. A model for scale up of family health innovations in low-income and middle-income settings: a mixed methods study. BMJ Open. 2012;2:e000987. doi: 10.1136/bmjopen-2012-000987. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Fixsen DL, Blase KA, Fixsen AA. Scaling effective innovations. Criminol Public Policy. 2017;16:487–99. [Google Scholar]
- 18.Ploeg J, Markle-Reid M, Davies B, et al. Spreading and sustaining best practices for home care of older adults: a grounded theory study. Implement Sci. 2014;9:162. doi: 10.1186/s13012-014-0162-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Rocker GM, Amar C, Laframboise WL, et al. Spreading improvements for advanced COPD care through a Canadian collaborative. Int J Chron Obstruct Pulmon Dis. 2017;12:2157–64. doi: 10.2147/COPD.S140043. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Gaziano TA, Galea G, Reddy KS. Scaling up interventions for chronic disease prevention: the evidence. Lancet. 2007;370:1939–46. doi: 10.1016/S0140-6736(07)61697-3. [DOI] [PubMed] [Google Scholar]
- 21.Community-based primary health care: research profiles. Ottawa: Canadian Institutes of Health Research; [accessed 2018 Apr. 26]. (modified 2018 May 8). Available: www.cihrirsc.gc.ca/e/50370.html#it12. [Google Scholar]
- 22.Wong ST, Langton JM, Katz A, et al. Promoting cross-jurisdictional primary health care research: implementing 12 community based primary health care teams across Canada. Prim Health Care Res Dev. doi: 10.1017/S1463423618000518. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Rogers EM. Diffusion of innovations. 4th ed. New York: Simon and Schuster; 1995. [accessed 2018 Oct. 2]. Available: https://books.google.ca/books?id=v1ii4QsB7jIC&printsec=frontcover#v=onepage&q&f=false. [Google Scholar]
- 24.Flay BR, Biglan A, Boruch RF, et al. Standards of evidence: criteria for efficacy, effectiveness and dissemination. Prev Sci. 2005;6:151–75. doi: 10.1007/s11121-005-5553-y. [DOI] [PubMed] [Google Scholar]
- 25.International Classification of Health Interventions (ICHI) Geneva: World Health Organization; [accessed 2018 Apr. 26]. (updated 2016 Aug. 18). Available: www.who.int/classifications/ichi/en/ [Google Scholar]
- 26.Deblaise D, Hernot X, Maurine P. A systematic analytical method for PKM stiffness matrix calculation. Proceedings 2006 IEEE International Conference on Robotics and Automation; 2006 May 15–19; Orlando (FL). pp. 4213–9. [Google Scholar]
- 27.Charmaz K, Belgrave LL. The Blackwell encyclopedia of sociology. Hoboken (NJ): John Wiley & Sons; 2015. [accessed 2018 Feb. 14]. Grounded theory. Available: http://onlinelibrary.wiley.com/doi/10.1002/9781405165518.wbeosg070.pub2/abstract. [Google Scholar]
- 28.Eisenhart M. Conceptual frameworks for research circa 1991: ideas from a cultural anthropologist; implications for mathematics education researchers. In: Underhill RG, editor. Proceedings of the Thirteenth Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education; 1991 Oct. 16–19; Blacksburg (VA). [accessed 2018 Jan. 18]. pp. 202–19. Available: http://nepc.colorado.edu/files/Eisenhart_ConceptualFrameworksforResearch.pdf. [Google Scholar]
- 29.Health indicators overview: questions and answers. Bethesda (MD): US National Library of Medicine; 2010. Jan 28, [accessed 2018 Jan. 18]. (updated 2018 July 3). Available: https://www.nlm.nih.gov/nichsr/healthindicators/Health_Indicators_Overview_Q_and_A.html. [Google Scholar]
- 30.Understanding capacity-building/capacity development: a core concept of development policy. Brussels: European Parliament; 2017. [accessed 2018 Jan. 18]. Available: www.europarl.europa.eu/thinktank/en/document.html?reference=EPRS_BRI(2017)599411. [Google Scholar]
- 31.Cooke J. A framework to evaluate research capacity building in health care. BMC Fam Pract. 2005;6:44. doi: 10.1186/1471-2296-6-44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.SAS/STAT® 142 user’s guide: the CLUSTER procedure. Cary (NC): SAS Institute; 2016. [accessed 2018 Apr. 26]. Available: https://support.sas.com/documentation/onlinedoc/stat/142/cluster.pdf. [Google Scholar]
- 33.Sokal R, Michener C. A statistical method for evaluating systematic relationship. Univ Kans Sci Bull. 1958;38:1409–38. [Google Scholar]
- 34.Fletcher J. Ethical approval for all studies involving human participants. CMAJ. 2015;187:91. doi: 10.1503/cmaj.141538. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Agarwal G, Angeles R, Pirrie M, et al. Effectiveness of a community paramedic-led health assessment and education initiative in a seniors’ residence building: the Community Health Assessment Program through Emergency Medical Services (CHAP-EMS) BMC Emerg Med. 2017;17:8. doi: 10.1186/s12873-017-0119-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Scott C, Barnes J, Mallard R, et al. Evaluating a pop-up health and community event in Lethbridge, Alberta, Canada. Proceedings of the Forty-fifth North American Primary Care Research Group meeting; 2017 Nov. 17–21; Montréal. Leawood (KS): North American Primary Care Research Group; [accessed 2018 Mar. 29]. Available: www.napcrg.org/Conferences/AnnualMeeting/EducationEvents/SearchEducationalSessions?m=6&s=21149. [Google Scholar]
- 37.Bradley EH. Community health workers are effective but how to scale-up? Am J Public Health. 2018;108:1129–30. doi: 10.2105/AJPH.2018.304602. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Rycroft-Malone J, Seers K, Titchen A, et al. What counts as evidence in evidence-based practice? J Adv Nurs. 2004;47:81–90. doi: 10.1111/j.1365-2648.2004.03068.x. [DOI] [PubMed] [Google Scholar]
- 39.Wiles R, Crow G, Pain H. Innovation in qualitative research methods: a narrative review. Qual Res. 2011;11:587–604. [Google Scholar]
- 40.Rogers EM. Diffusion of innovations. 5th ed. New York: Free Press; 2003. [Google Scholar]
- 41.Rapport F, Clay‐Williams R, Churruca K, et al. The struggle of translating science into action: foundational concepts of implementation science. J Eval Clin Pract. 2018;24:117–26. doi: 10.1111/jep.12741. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Wainwright DW, Waring TS. The application and adaptation of a diffusion of innovation framework for information systems research in NHS general medical practice. J Inf Technol. 2007;22:44–58. [Google Scholar]
- 43.Bobrow K, Farmer A, Cishe N, et al. Using the Medical Research Council framework for development and evaluation of complex interventions in a low resource setting to develop a theory-based treatment support intervention delivered via SMS text message to improve blood pressure control. BMC Health Serv Res. 2018;18:33. doi: 10.1186/s12913-017-2808-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Harting J, Rutten GM, Rutten ST, et al. A qualitative application of the diffusion of innovations theory to examine determinants of guideline adherence among physical therapists. Phys Ther. 2009;89:221–32. doi: 10.2522/ptj.20080185. [DOI] [PubMed] [Google Scholar]
- 45.Taylor C, Coffey A. Innovation in qualitative research methods: possibilities and challenges. Cardiff UK: Cardiff University; 2008. (School of Social Sciences Working Papers series vol 121). [Google Scholar]
- 46.Ilott I, Gerrish K, Pownall S, et al. Exploring scale-up, spread, and sustainability: an instrumental case study tracing an innovation to enhance dysphagia care. Implement Sci. 2013;8:128. doi: 10.1186/1748-5908-8-128. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Fixsen AAM. Defining scaling up across disciplines: an annotated bibliography. Portland (OR): Portland State University; 2009. [accessed 2018 Sept. 28]. Available: https://pdfs.semanticscholar.org/57fc/57337970444b7bfc536adee7d0791a36471b.pdf. [Google Scholar]
- 48.Indig D, Lee K, Grunseit A, et al. Pathways for scaling up public health interventions. BMC Public Health. 2017;18:68. doi: 10.1186/s12889-017-4572-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Forman J, Heisler M, Damschroder LJ, et al. Development and application of the RE-AIM QuEST mixed methods framework for program evaluation. Prev Med Rep. 2017;6:322–8. doi: 10.1016/j.pmedr.2017.04.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89:1322–7. doi: 10.2105/ajph.89.9.1322. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.