Assessment of scalability of evidence-based innovations in community-based primary health care: a cross-sectional study

Abstract

Background:

In 2013, the Canadian Institutes of Health Research funded 12 community-based primary health care research teams to develop evidence-based innovations. We aimed to explore the scalability of these innovations.

Methods:

In this cross-sectional study, we invited the 12 teams to rate their evidence-based innovations for scalability. Based on a systematic review, we developed a self-administered questionnaire with 16 scalability assessment criteria grouped into 5 dimensions (theory, impact, coverage, setting and cost). Teams completed a questionnaire for each of their innovations. We analyzed the data using simple frequency counts and hierarchical cluster analysis. We calculated the mean number and standard deviation (SD) of innovations that met criteria within each dimension that included more than 1 criterion. The analysis unit was the innovation.

Results:

The 11 responding teams evaluated 33 evidence-based innovations (median 3, range 1–8 per team). The innovations focused on access to care and chronic disease prevention and management, and varied from health interventions to methodological innovations. Most of the innovations were health interventions (n = 21), followed by analytical methods (n = 4), conceptual frameworks (n = 4), measures (n = 3) and strategies to build research capacity (n = 1). Most (29) met criteria in the theory dimension, followed by impact (mean 22.3 [SD 5.6] innovations per dimension), setting (mean 21.7 [SD 8.5]), cost (mean 17.5 [SD 2.1]) and coverage (mean 14.0 [SD 4.1]). On average, the innovations met 10 of the 16 criteria. Adoption was the least assessed criterion (n = 9). Most (20) of the innovations were highly ranked for scalability.

Interpretation:

Scalability varied among innovations, which suggests that readiness for scale up was suboptimal for some innovations. Coverage remained largely unaddressed; further investigation of this critical dimension is necessary.

The development and implementation of evidence-based innovations have been strongly incentivized as part of primary care reform in Canada.¹ The research focus has been on generating enough evidence of sufficient rigour to qualify innovations as evidence-based.^2–4 However, there is a growing gap between development of evidence-based innovations as research projects and their widespread implementation as standard care.^5–7 Scale up or spread is a form of knowledge translation that aims to increase the reach and adoption of innovations.^8–13 Successful scale up follows several steps: 1) assessment of scalability (potential for scale up) of the innovation (e.g., effectiveness, cost-effectiveness), 2) development of a scale-up strategy, 3) strategy implementation, 4) strategy evaluation and 5) assuring sustainability.^13–16 Rigorous evaluation of scale up includes measuring coverage (number of people, organizations or systems that adopt the innovation over the number targeted).^2,17 In Canada, however, there are few systematic efforts to facilitate and support scale up and few theoretical, conceptual or practical frameworks for guiding the first step, scalability assessment.^13,14 This leads to a lag between evidence generation and scale up, and denies patients and communities access to effective services.^7,18–20

In 2013, 12 research teams were funded by the Canadian Institutes of Health Research to conduct cross-jurisdictional research in improving access to community-based primary health care for vulnerable populations, and chronic disease prevention and management.²¹ As a funding condition, teams were required to collaborate in sharing their findings and lessons learned, build capacity and plan for scale up.²² In the present study, we aimed to explore the scalability of the teams’ evidence-based innovations.

Methods

Study design

In this cross-sectional study, we invited the 12 research teams to rate the scalability of their evidence-based innovations. An evidence-based innovation was defined as a program, model, approach, tool, instrument, indicator, algorithm, service, idea, policy or practice whose evidence base has been established.^23,24

Setting and participants

We conducted the study between August and December 2017 together with the teams. The teams covered all of Canada except for 1 province (Saskatchewan) and 1 territory (Yukon Territory).^21,22 They included over 100 stakeholders in community-based primary health care.^21,22 Their research was programmatic, i.e., each team worked on varied but conceptually linked projects bound together by the themes of access to care for vulnerable populations, and chronic disease prevention and management.^21,22

Data collection

We created a 1-page self-administered questionnaire containing the key elements for assessing the scalability of the teams’ evidence-based innovations (Appendix 1, available at www.cmajopen.ca/content/6/4/E520/suppl/DC1). We designed the questionnaire based on the results of a systematic review of scale-up strategies in primary care² and the recommendations of 2 scale-up guides.^13,14 The teams were invited to rate the scalability of their evidence-based innovations and were asked for details about their team and the name, type and aim of the innovation. The questionnaire included 16 criteria for assessing scalability, grouped into 5 dimensions: 1) the theory used to develop the innovation, 2) impact (e.g., has efficacy been assessed), 3) likely coverage of the innovation (e.g., has adoption been assessed), 4) alignment of the innovation with the setting (e.g., is it compatible with similar interventions in the same setting) and 5) cost (e.g., is scale up affordable). For each criterion, there were 5 response options: 1) “Yes” (criterion was assessed), 2) “No” (criterion was not assessed), 3) “UE” (criterion was under evaluation), 4) “NP” (criterion assessment was not planned) and 5) “NA” (not applicable). Space was provided for additional comments on each criterion. The questionnaire was piloted by 1 of the 12 teams (S.T.W.’s team) and was revised to improve understanding of the instructions and examples of evidence-based innovations. All teams were asked to return their completed responses by email. A maximum of 3 reminder messages were sent no later than 3 weeks after the first message. A team could rate more than 1 of their innovations (1 questionnaire each).

Data analysis

Two of the authors (A.B.C. and H.T.V.Z.) independently classified the innovations into 1 of 5 mutually exclusive types according to predefined themes. Any discrepancies were resolved through consensus with the senior author (F.L.). The themes corresponded to the following types of evidence-based innovations:

• Health intervention: an act performed for, with or on behalf of a person or a population to improve, maintain, promote or modify health functioning or health conditions (e.g., preventive strategy, screening program, decision aid)²⁵

• Analytical method: a generic or systematic process combining the scientific method with the use of a formal process to solve a research problem^26,27

• Conceptual framework: a set of concepts or abstractions linked and arranged rationally according to their relevance to a common theme²⁸

• Measure: any instrument, measure or indicator²⁹

• Strategy to build research capacity: a process of individual or institutional development that leads to a greater ability to perform useful research.^30,31

Team respondents were involved in all steps of the study. Together, we grouped the response options for the 16 scalability assessment criteria into 3 categories: criterion met (responses corresponding to “criterion was assessed” and “criterion was under evaluation”), criterion not met (responses corresponding to “criterion was not assessed” and “criterion assessment was not planned”) and not applicable. Thus, each innovation was scored on the number of criteria met. Then we ranked the 16 criteria according to the number of innovations that met each of these criteria. For the dimensions that include more than 1 criterion, we calculated the mean number and standard deviation (SD) of innovations per dimension for which criteria were met. We used analysis of variance to compare the number of criteria met per innovation.

We analyzed the data using simple frequency counts and hierarchical cluster analysis. The unit of analysis was the innovation. As we had very little missing data, no specific missing-case analyses were conducted. We analyzed the data using SAS version 9.4 (SAS Institute). We conducted a hierarchical cluster analysis among the innovations using the SAS CLUSTER procedure.³² This allowed us to group innovations into the most homogeneous clusters possible, based on the number of criteria met. Thus, the hierarchical clustering analysis was aimed to create clusters of innovations but not to analyze existing clusters. The objective was to rank innovations in order of their scalability. We used the average linkage method, whereby the distance between 2 clusters is the average of the distances of all pairs of innovations, 1 in each cluster. ^32,33 We estimated the optimal number of clusters using the pseudo F and pseudo t² statistics.³²

Ethics approval

Our study assessed research innovations and did not collect data for human participants; thus, ethics review was not needed.³⁴

Results

Participants

Eleven of the 12 teams reported at least 1 evidence-based innovation (Figure 1). One team did not respond owing to lack of time and capacity. We received information for 33 innovations (median 3, range 1–8 per team). As shown in Table 1, 23 innovations focused on chronic diseases and/or access to resources (e.g., health care services, social services and complex care needs).

Figure 1:

Flow chart showing identification of evidence-based innovations.

Table 1:

Characteristics of the 33 evidence-based innovations reported by 11 teams

Type of innovation	Scalability ranking*	No. of scalability assessment criteria met†	Reported aim(s) of innovation
Health intervention	High	16	Increase access to primary health care for people/populations that are underserved by, and struggle to connect with, traditional primary health care services; improve coordination and collaboration of primary health care service providers when providing care
Health intervention	High	14	Multifaceted interprofessional community-based self-management program for older adults with diabetes and multiple chronic conditions
Health intervention	High	14	Introduce innovative team work for the management of patients with multimorbidity
Health intervention	High	14	Enhance the new patient’s capacity to access the clinic and understand how it functions
Health intervention	High	13	Improve diabetes prevention and management by leveraging existing resources
Health intervention	High	13	Interprofessional community-based self-management program for older adults with vascular conditions and multiple chronic conditions
Health intervention	High	13	Actionable information on primary care for key stakeholders; sustainable performance portrait (through ongoing funding); innovation through information to stakeholders
Health intervention	High	13	Reduce the impact of vascular disease on the health of Canadians
Health intervention	High	12	Integrate chronic disease prevention and management by a team in primary care practice
Health intervention	High	12	Develop knowledge, skills and abilities to work with primary care practices to support people experiencing social barriers in reaching community health and social resources
Health intervention	High	11	Identify patients at increased risk for breast and colorectal cancer and personalize screening and management; help patients with breast cancer make treatment decisions
Health intervention	High	11	Integrated approach for chronic diseases
Health intervention	High	11	Improve communication between primary care providers and cancer specialists; improve continuity of care among patients with cancer
Health intervention	High	11	Type of patient referrals, navigation services provided, community health and social resource needs
Health intervention	High	11	Provide rural remote primary care
Health intervention	High	10	Toolkit to support caregivers of older adults with dementia and multiple chronic conditions with changes they are experiencing
Health intervention	Medium	8	Ensure meaningful engagement and participation of people with lived experience in the research process, and ensure that people with lived experience are adequately compensated for their time and energy devoted to the research
Health intervention	Medium	5	Explore how people might use publicly reported primary care information through a series of citizen-patient deliberations in 3 Canadian provinces
Health intervention	Medium	4	Lead to awareness among allied health care professionals and policy shifts in health care systems regarding mental health care for youth and support for caregivers
Health intervention	Low	1	Present the clinical cancer system and experiences of patients with breast and colorectal cancer in Canada through synthesis maps to be used as knowledge-translation tools
Health intervention	Low	0	Profile Canadian initiatives aimed at improving continuity of care between primary care providers and cancer specialists
Analytical method	High	12	Piloted a practice-based automated patient survey system by telephone or email with consenting patients
Analytical method	Medium	9	Comparability of health administrative data indicator definitions across provinces
Analytical method	Medium	6	Gather narratives/stories of youth mental health journeys from multivocal perspectives
Analytical method	Medium	5	Use analytical modelling to support children and youth with mental health conditions
Conceptual framework	High	13	Enable health care providers to implement integrated care for people with complex health needs
Conceptual framework	High	10	Inform implementation of patient-centred interventions to improve care of patients with multimorbidity
Conceptual framework	Medium	8	Support innovation and transformation in First Nations health
Conceptual framework	Medium	6	Understand mental health from a First Nations perspective
Measure	Medium	9	Identify people with multimorbidity in primary care
Measure	Medium	8	Assess the integration, coordination and transitions of care
Measure	Medium	6	Assess the correlations of sex and gender with patient outcomes
Research capacity	High	10	Build research capacity

^*Scalability ranking groups are based on hierarchical cluster analysis.

^†Range 0–16.

Types of evidence-based innovations

The majority (21) of the evidence-based innovations were health interventions (e.g., a community partnership program aiming to reduce the impact of vascular disease on the health of Canadians³⁵), followed by analytical methods (4 innovations) (e.g., piloting an automated practice-based patient survey system by telephone or email), conceptual frameworks (4 innovations) (e.g., a guide to implementing integrated care), measures (3 innovations) (e.g., a 21-item measure for identifying people with multimorbidity in primary care) and strategies to build research capacity (1 innovation).

Scalability assessment dimensions

Use of a theory to develop the innovation (1 criterion) was the dimension in which scalability assessment criteria were most often met (n = 29) (e.g., the Consolidated Framework for Implementation Research), followed by impact (mean 22.3 [SD 5.6] innovations per dimension), setting (mean 21.7 [SD 8.5]), cost (mean 17.5 [SD 2.1]) and coverage (mean 14.0 [SD 4.1]) (Table 2). Within the coverage dimension, the criteria of reach, adoption and maintenance were frequently reported as not applicable (n = 12, 12 and 14, respectively) (Supplementary Table A1, Appendix 1), with the most commonly reported reason being that the innovation was “not an intervention.”

Table 2:

Number of evidence-based innovations that met scalability assessment criteria

Assessment dimension/criteria	No. (%) of innovations that met criteria	Mean* no. of innovations that met criteria per dimension ± SD
Use of theory: Evidence-based innovation developed with theory	29 (88)	NA
Impact		22.3 ± 5.6
Acceptability	22 (67)
Feasibility	26 (79)
Adaptability	19 (58)
Efficacy	13 (39)
Effectiveness	27 (82)
Results documented	27 (82)
Setting		21.7 ± 8.5
Implemented in setting comparable to target setting	28 (85)
Compatibility with similar evidence-based innovations in target settings	12 (36)
Consistency with policy directives	25 (76)
Cost		17.5 ± 2.1
Cost-effectiveness	16 (48)
Resources needed for scale up (affordability)	19 (58)
Coverage		14.0 ± 4.1
Reach (numerator and denominator)	14 (42)
Adoption (numerator and denominator)	9 (27)
Fidelity	19 (58)
Maintenance	14 (42)

Note: NA = not applicable, SD = standard deviation.

^*Sum of the number of innovations that met the criteria in each dimension divided by the total number of criteria in that dimension.

On average, 10 (SD 4) of the 16 scalability assessment criteria were met by the 33 innovations. Adoption was the least assessed criterion (9 innovations). The number of criteria that were met in the coverage dimension varied with the type of innovation (p = 0.005), with health interventions likely to meet the most criteria.

Ranking for scalability

Using hierarchical cluster analysis, we classified the 33 innovations into 3 groups (pseudo F = 73.5, pseudo t² = 8.0): those whose scalability was ranked as high (n = 20), those whose scalability was ranked as medium (n = 11) and those whose scalability was ranked as low (n = 2) (Table 1). The mean number of scalability assessment criteria met for the 3 groups was 12 (SD 2), 7 (SD 2) and 1 (SD 1), respectively. A high ranking indicated that the team had collected diverse important information relevant to making a decision about the scale up of the innovation. Nine of the 11 responding teams ranked at least 1 of their innovations as having high potential for scale up (median 1, range 0−5 per team). Most (16) of the 20 high-ranking innovations were health interventions.

Interpretation

In this study, we explored the scalability of a large sample of evidence-based innovations in Canadian community-based primary health care. The majority of innovations were health interventions, and most ranked high for scalability. However, few of the innovations met the scalability assessment criteria relating to coverage, a dimension essential for scalability. These findings lead us to make the following observations.

First, the innovations that ranked highest for scalability were health interventions. For example, the pop-up health and community service event met all of the scalability assessment criteria.³⁶ As the innovation had already been implemented and assessed in several different settings (e.g., a seniors’ centre, the Indigenous Early Intervention Organization, an elementary school), the team was likely to have collected a variety of relevant data. A crucial element of successful scale up is user-informed assessment of the environment to determine whether there is a fit between the target units and the innovation.^16,37 A danger in health care system reform is the rolling-out of innovations as population-wide “one-size-fits-all” solutions. This project had clearly based its scaled-up implementation on individual and community-based needs.^1,36

Second, innovations that were not health interventions were overall ranked lower in scalability. The diversity of innovations developed by the teams (from health interventions to methodological innovations) reflects the inherent complexity of community-based primary health care research, not all of which can be appropriately or easily scaled up. Teams whose innovations consisted of a conceptual framework or an analytical method, for example, were less concerned about widespread implementation of their innovations as they were not necessarily designed to directly improve health functioning or alleviate conditions.³⁸ In addition, the diffusion of methodological innovation often takes a more complex, intuitive and organic path that is more horizontal than top-down. ^8,39,40 However, there is value in assessing, adapting and scaling up methodological innovations for wider use by researchers,^39,41–44 and there are too few opportunities for researchers to develop and experiment with methods for addressing social and clinical research questions.^39,45 The usefulness of all scaled-up innovations will ultimately depend on the strength of the methodology developed.

Third, we found that coverage was the dimension in which scalability assessment criteria were least likely to be met, and adoption was the least assessed criterion. This may be in part because some innovations were not yet fully implemented or were designed to remain local rather than scalable. Lack of scalability frameworks to support research design and lack of sufficient time and resources could be other reasons. Moreover, some of the concepts relating to the science of scale up (e.g., adoption, adaptation, maintenance, spread and scale up itself) are as yet ill-defined and undertheorized in knowledge translation.⁴⁶ Some of these concepts were unfamiliar to some of the stakeholders responding to our questionnaire. This highlights the need to refine and standardize terminology relating to scale-up science.^2,47

Finally, it is difficult to compare our results with the literature, as we found few studies that analyzed primary health care innovations using scalability assessment criteria. The authors of a study assessing scaled-up public health interventions found that 25% of interventions had not been trialled, and, in 5%, efficacy had not been assessed, but they did not address coverage.⁴⁸ The authors of a systematic review on scaling up evidence-based innovations in primary care noted that coverage was least likely to be assessed.² Coverage of targeted users or settings is at the heart of scalability.^2,13,17 For innovations to have a substantial impact, they need to be adopted by a large enough population over a sustained period.¹³ Developers of innovations can evaluate their programs using the Reach, Effectiveness, Adoption, Implementation, and Maintenance Qualitative Evaluation for Systematic Translation (RE-AIM QuEST) framework to specifically explore the coverage potential of their innovations.^49,50

Limitations

First, in the interests of brevity, our questionnaire did not adequately explain every dimension of assessing scalability. Two teams contacted us for further explanation. Although the questionnaire was not validated, it was created through a group process (with 3 teams) and was pretested with at least 3 innovations. Second, we collected data on the existence of assessment criteria but not on the results of those assessments. Thus, based on our data, we cannot confidently label the innovations as scalable or not. However, our data clearly show that some teams collected important information that would facilitate scale up of their innovations. Moreover, feedback from the team members indicated that they found the experience constructive and informative for the purposes of future scale-up planning. Third, the questionnaires were completed by team stakeholders who, although best qualified to answer detailed questions about their projects, were also researchers in the study. This may have reduced objectivity. Also, all teams had end-user partners, and if these partners had completed questionnaires as well, they would have added an important perspective on the project’s scalability. Finally, scores obtained with the hierarchical cluster analysis are specific to the innovations used, which does not allow for generalization of results. As a first exercise to rank the scalability of evidence-based innovations in community-based primary health care in Canada, these findings will enable us to refine the language of scaling up, adapt the scalability assessment criteria, improve our questionnaire and validate it for future studies.

Conclusion

We explored scalability assessment criteria among a large sample of evidence-based innovations in Canadian community-based primary health care. Overall, teams had important information about their innovations that they could use to plan scale up. However, coverage is a critical dimension that remains largely unaddressed. Our findings provide concrete data on evaluating innovations for larger-scale use and contribute important new understandings of scalability assessment with relevance to a broad group of stakeholders. The questionnaire itself served as a knowledge-translation tool, motivating teams to talk with their research communities and decision-makers about scalability. Finally, our findings inform stakeholders about how and why scalability should be evaluated and reported as an outcome measure in project design and grant submission.