Assessing the scalability of innovations in primary care: a cross-sectional study

Abstract

Background:

Canadian health funding currently prioritizes scaling up for evidence-based primary care innovations, but not all teams prepare for scaling up. We explored scalability assessment among primary care innovators in the province of Quebec to evaluate their preparedness for scaling up.

Methods:

We performed a cross-sectional survey from Feb. 18 to Mar. 18, 2019. Eligible participants were 33 innovation teams selected for the 2019 Quebec College of Family Physicians’ Symposium on Innovations. We conducted a Web-based survey in 2 sections: innovation characteristics and the Innovation Scalability Self-administered Questionnaire. The latter includes 16 criteria (scalability components) grouped into 5 dimensions: theory (1 criterion), impact (6 criteria), coverage (4 criteria), setting (3 criteria) and cost (2 criteria). We classified innovation types using the International Classification of Health Interventions. We performed a descriptive analysis using frequency counts and percentages.

Results:

Out of 33 teams, 24 participated (72.7%), with 1 innovation each. The types of innovation were management (15/24), prevention (8/24) and therapeutic (1/24). Most management innovations focused on patient navigation (9/15). In order of frequency, teams had assessed theory (79.2%) and impact (79.2%) criteria, followed by cost (77.1%), setting (59.7%) and coverage (54.2%). Most innovations (16/24) had assessed 10 criteria or more, including 10 management innovations, 5 prevention innovations and 1 therapeutic innovation. Implementation fidelity was the least assessed criterion (6/24).

Interpretation:

The scalability assessments of a primary care innovation varied according to its type. Management innovations, which were the most prevalent and assessed the most scalability components, appear to be most prepared for primary care scale-up in Canada.

Primary care refers to care received in the context of patients’ first contact with the health care system.^1–3 In Canada, primary care is at the centre of major reforms.⁴ Canadian health funding currently prioritizes the large-scale implementation of practices or products perceived as new in terms of a decision to adopt (“innovation” ^5,6), that is, the scaling up of effective primary care initiatives nation-wide.^4,7–11 The rationale is that scaling up primary care innovations will reduce waste in health services, reduce health inequities and improve the health of Canadians. ^4,12 Thus, there is a need to identify evidence-based primary care innovations that could be successfully expanded to reach more patients.^8,13,14 The first step is to find out whether such innovations have been planned with scaling up in mind (i.e., whether their producers have assessed the components of their scalability).

In knowledge translation (KT) or implementation science (both hereafter referred to as KT), the differences between “scaling up,” “scaling out,” “scaling deep,” “scaling” and “spread” are nuanced.^14–16 Here, we define the process of “scaling up” or “scale up” as “deliberate efforts to increase the impact of successfully tested health innovations so as to benefit more people and to foster policy and program development on a lasting basis.”⁶

To be successful, scaling up should follow a number of steps.^6,8,17–20 The scalability assessment is the preliminary step, or the evaluation of the “ability of a health innovation shown to be efficacious on a small scale and/or under controlled conditions to be expanded under real-world conditions to reach a greater proportion of the eligible population, while retaining effectiveness.”²¹ Scalability assessments are often overlooked,^14,22 with unfortunate results, such as the replicating of harms at scale.^8,14,23 To be scalable, an innovation should meet certain minimum criteria responding to the essential components of scalability,^{6,8,17,21,24,25} such as implementation fidelity.^26,27 In Canada, there are few systematic guides to assessing scalability, and policy-makers face a predicament when choosing between innovations to scale up in primary care.^8,13

As the voice of family medicine in Canada, The College of Family Physicians of Canada promotes primary care innovation to improve the health of Canadians.²⁸ As part of this mission, the Quebec chapter of the college held a Symposium on Innovations on May 31, 2019, in Montréal to catalyze the scale-up of primary care innovations across the province by gathering together innovation teams, patients, citizens, clinicians and decision-makers.²⁹ As part of the preparation for this symposium, we sought to explore scalability assessment among these primary care innovators in the province of Quebec to evaluate their preparedness for scaling up province-wide.

Methods

Study design

We performed a cross-sectional Web-based survey and used the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) to report our study.³⁰

Participants

Eligible participants were the 33 teams selected by the Quebec College of Family Physicians to participate in the college’s Symposium on Innovations. The innovation teams were recruited for the symposium by the Quebec College of Family Physicians in collaboration with the Quebec Strategy for Patient-Oriented Research (SPOR) Support for People and Patient-Oriented Research and Trials (SUPPORT) Unit,³¹ Réseau-1 Québec³² and McGill University³³ through emails, social media advertising, snowball sampling and volunteer sampling. Selected innovations met the following criteria: has a potentially important impact on the organization of primary care in the province of Quebec, is aligned with the vision of the Patient’s Medical Home (a family practice that offers readily accessible patient-centred, team-based primary care)³⁴ and contributes to a culture of continuous quality improvement. Each team was represented by the team member (e.g., a coordinator, clinician or researcher) responsible for communicating with the symposium coordinator (M.-J.C.).

Questionnaire development

We used a 1-page self-administered questionnaire developed by the first author (A.B.C.) and colleagues.⁸ It was based on the recommendations of a systematic review on scaling-up strategies in primary care¹⁴ and 2 scaling-up frameworks.^6,17 The questionnaire included 2 sections: characteristics of the innovation (e.g., type and aim) and the Innovation Scalability Self-administered Questionnaire (ISSaQ) (Appendix 1, available at www.cmajopen.ca/content/8/4/E613/suppl/DC1).⁸

The ISSaQ aims to assess the scalability of primary care innovations. It includes 16 criteria (representing scalability components) in 5 dimensions: theory used in developing the innovation (1 criterion), impact of the innovation (6 criteria), likely coverage (4 criteria), setting (3 criteria) and cost (2 criteria). ⁸ For each criterion, there are 5 possible responses: “Yes,” criterion assessed; “No,” criterion not assessed; “UE,” criterion under evaluation; “NP,” criterion assessment not planned; and “NA,” not applicable. Space was provided for additional comments on each criterion, including rationale for choosing “not applicable.”

For the present study, we translated the original questionnaire into French using the cross-cultural adaptation process.³⁵ Two French mother-tongue translators (including an expert in KT), produced 2 independent translations. A.B.C. checked their accuracy and resolved discrepancies through consensus with the translators. Two English mother-tongue translators (including an expert in KT) translated the first French draft back into English. All differences with the original were discussed by the research group, and the questionnaire was modified for comprehensibility and conceptual equivalence.

Our questionnaire was the first part of a 3-step validation process that involves testing the scalability components,⁸ reviewing all available scalability assessment tools³⁶ and conducting a multistakeholder consensus exercise.

Data collection

A.B.C. and J.M. created a Web-based survey for the questionnaire using the SurveyMonkey platform. It solicited a single mandatory response for each scalability criteria. Eight authors (4 male and 4 female), representing key stakeholder groups (family physicians [J.M., M.D.P., E.B. and F.L.], a health care manager [M.-J.C.], a scientific coordinator [H.T.V.Z.], trainees [A.B.C., J.M. and A.G.] and researchers [A.B.C., J.M., M.D.P., A.G. and F.L.]) pretested the 10-minute Web-based survey, refining its structure and improving the explanations of each criteria. The corresponding members of the 33 teams were invited by email to participate in the survey (Appendix 2, available at www.cmajopen.ca/content/8/4/E613/suppl/DC1). Two reminder emails were sent and the corresponding team members were given an additional day to complete the survey. The survey was open from Feb. 18 to Mar. 18, 2019.

We offered no incentive for completing the survey. We asked for the name and email of each team wishing to receive feedback on our study. We collected no further information on the teams. Reasons for nonresponse were determined through email exchanges with the corresponding members of the team.

Statistical analysis

Two authors (A.B.C. and L.K.) independently classified the innovations using the International Classification of Health Interventions. ^37,38 This classification includes 3 axes: target (the entity on which the action is carried out); action (the deed done by an actor to the target); and means (the processes and methods by which the action is carried out). For this survey, the action axis was the most appropriate for categorizing innovations that act to change systems or behaviours. Within this axis are 6 categories of innovation: management, prevention, therapeutic, diagnostic, other (i.e., not elsewhere classified) and unspecified action. Discrepancies were resolved through consensus between A.B.C., L.K. and a third author (J.M. or H.T.V.Z.).

As suggested by key stakeholders,⁸ we grouped the response options for the 16 scalability criteria into 3 categories: assessed (i.e., responses corresponding to “criterion assessed” and “criterion under evaluation”), not assessed (i.e., responses corresponding to “not assessed” and “not planned”) and not applicable. Each criterion assessed scored 1 point and 0 points otherwise. The “not applicable” response was considered equal to “criterion not assessed,” because few cases were reported and no adequate rationale was provided. The highest possible assessment score for an innovation was 16 points. High scores indicated the team had assessed more scalability components of their innovation. Based on previous findings,⁸ we also grouped innovations as follows: those whose scalability assessment ranked as high (i.e., innovations assessed ≥ 10 criteria), medium (i.e., innovations assessed 4–9 criteria) and low (i.e., innovations assessed ≤ 3 criteria).

We performed a descriptive analysis using simple frequency counts and percentages. The individual innovation was the unit of analysis. Within the scalability dimensions, we calculated the arithmetic mean of the percentages of innovations that assessed the criteria. Data were analyzed using R software (version 3.5.1).

Ethics approval

Ethics approval was waived by the ethics board of the Centre intégré universitaire de santé et de services sociaux (CIUSSS) de la Capitale-Nationale as our survey assessed research innovations and did not collect data on human participants.³⁹

Results

Out of 33 contacted teams, 24 participated (72.7% response rate) with 1 innovation each. All questionnaires returned by these 24 teams were complete. The 9 other teams did not complete the questionnaire because of lack of time or resources (3 teams), lack of results (1 team) and unknown reasons (5 teams).

Types of innovation

Types of innovation were in the management (15/24), prevention (8/24) and therapeutic (1/24) categories (Table 1). Descriptions of each innovation can be found in Appendix 3, available at www.cmajopen.ca/content/8/4/E613/suppl/DC1.

Table 1:

Description of innovations from 24 teams recruited for the Symposium on Innovations

Category of innovation according to the International Classification of Health Interventions*		Scalability ranking; no. of innovations†
Type	Subtype	High (10–16 criteria assessed)	Medium (4–9 criteria assessed)	Low (0–3 criteria assessed)	Total
Management, n = 15	Navigating	8	0	1	9
	Collaborating	1	2	1	4
	Prescription	1	1	0	2
Prevention, n = 8	Capacity building	2	1	3
	Public health surveillance	2	1	0	3
	Marshalling health services or health-related services	0	1	0	1
	Restrictions on advertising, promotion or sponsorship of products	1	0	0	1
Therapeutic, n = 1	Assisting or promoting exercise	1	0	0	1
Total		16	6	2	24

^*We found no innovation in the other categories (e.g., diagnostic).

^†Calculated using team responses to the Innovation Scalability Self-administered Questionnaire.⁸

Management innovations focused on patient navigation (i.e., assisting patients to identify or to access appropriate services and resources; 9/15), followed by interprofessional collaboration (i.e., working together with patients, health care providers and other stakeholders; 4/15) and prescription services (i.e., recommendation to obtain or pursue a specified health intervention; 2/15).

Preventive innovations focused on capacity building (3/8), followed by public health surveillance (3/8), marshalling health services (1/8), and restrictions on advertising, promotion or the sponsorship of products (1/8).

The only therapeutic innovation focused on assisting or promoting an exercise (cognitive, physical or sensory).

Scalability dimensions

In order of frequency, most innovations assessed theory (79.2%) and impact (79.2%) criteria, followed by cost (77.1%), setting (59.7%) and coverage (54.2%) (Table 2). On average, 11 of the 16 scalability criteria were assessed by the 24 innovation teams. Implementation fidelity was the least assessed criterion (6/24).

Table 2:

Description of scalability criteria among 24 innovations

Assessment dimension (no. of criteria, arithmetic mean* of the percentages of innovations that assessed the criteria of each dimension)	Criterion†	No. (%) of innovations
		Criterion assessed	Criterion not assessed	Criterion not applicable
Use of theory (1 criterion, 79.2%)	Innovations developed with theory	19 (79.2)	5 (20.8)	0
Impact assessments (6 criteria, 79.2%)	Acceptability	21 (87.5)	2 (8.3)	1 (4.2)
	Feasibility	21 (87.5)	2 (8.3)	1 (4.2)
	Efficacy‡	19 (79.2)	4 (16.7)	1 (4.2)
	Adaptability	13 (54.2)	8 (33.3)	3 (12.5)
	Effectiveness	22 (91.7)	2 (8.3)	0
	Results documented	20 (83.3)	4 (16.7)	0
Cost assessments (2 criteria, 77.1%)	Cost-effectiveness‡	16 (66.7)	7 (29.2)	1 (4.2)
	Resources needed for the scaling up (affordability)	21 (87.5)	2 (8.3)	1 (4.2)
Setting assessments (3 criteria, 59.7%)	Implemented in setting comparable to target setting	14 (58.3)	10 (41.7)	0
	Compatibility with similar innovations in target settings‡	11 (45.8)	11 (45.8)	2 (8.3)
	Consistency with policy directives	18 (75.0)	6 (25.0)	0
Coverage assessments (4 criteria, 54.2%)	Reach	11 (45.8)	9 (37.5)	4 (16.7)
	Adoption‡	14 (58.3)	8 (33.3)	2 (8.3)
	Fidelity	6 (25.0)	16 (66.7)	2 (8.3)
	Maintenance	21 (87.5)	3 (12.5)	0

^*Arithmetic mean = sum of the percentages of innovations that assessed the criteria in each dimension divided by the total number of criteria in that dimension, e.g., for the dimension coverage: 54.2% = (45.8% + 58.3% + 25.0% + 87.5%) ÷ 4.

^†The assessment data come from the teams’ responses to the Innovation Scalability Self-administered Questionnaire.⁸

^‡Percentages in the row may add up to ± 100% because of rounding error.

Rankings for scalability assessment

Scalability assessment was ranked as high for 16 innovations, medium for 6 innovations and low for 2 innovations (Table 1). High-ranking innovations were in the management (n = 10), preventive (n = 5) and therapeutic (n = 1) categories. The 10 high-ranking management innovations included 8 in patient navigation, 1 in interprofessional collaboration and 1 in prescription.

Interpretation

Our study explored scalability assessments of 24 primary care innovations in the province of Quebec. Management innovations, mostly focused on patient navigation and inter-professional collaboration, were those that ranked highest for scalability assessment and thus appear to be most prepared for scale-up. One innovation, a collaborative electronic prescription service that protects patient data (PrescribeIT), had assessed all 16 scalability criteria.^40,41 Already implemented in communities in Ontario, Alberta, New Brunswick and Saskatchewan, it seemed ready for rollout to additional provinces, such as Quebec. Although about half of all innovations addressed all scalability dimensions, implementation fidelity is a critical component that remained largely unaddressed. These findings lead us to make the following observations.

First, our sample of innovations reflects the evolution of primary care philosophy toward patient-oriented approaches to care delivered by multidisciplinary health care teams.^4,42–46 It also reflects current health funding priorities in Canada, which promote the scaling up of management and preventive interventions over types that improve health more directly.^7–10 These funding priorities provide motivation and resources to focus both on management solutions and on scaling up. In addition, Quebec is one of the provinces that has made the most efforts to scale up primary care innovations province-wide. ^34,47 In our study, it is therefore no surprise that management innovations predominated and that they were those that integrated scalability assessments the most. Provincial priorities for health care system reform may have made producers more aware of the relevance of generating evidence and the degree of rigour required to qualify these types of innovations as scalable. Thus, our findings capture the evolution of family medicine in Quebec and could catalyze the effective scaling up of management innovations in primary care.

Second, teams either considered implementation fidelity irrelevant to their innovations or did not know what it was. A previous study found that for most primary care innovations, implementation fidelity was rated “not applicable.”⁸ In spite of little attention being paid to implementation fidelity among our participants, when an innovation is not implemented as was originally intended, it is less likely to be effective, potentially leading to faulty conclusions about its potential for scale-up.^27,48,49 Achievement of high implementation fidelity is one of the best ways of replicating the success of the original research and is associated with better health outcomes.^50–53 Adaptability is an important component of a scalable innovation, but adaptability also makes implementation fidelity more difficult to achieve.^53,54 Although a strong focus on implementation fidelity may seem to contradict the need for adaptability, some authors have argued that adapting an innovation too drastically may actually decrease innovation effectiveness.⁵⁵ Our scalability assessment tool reflected the goal of achieving a balance between implementation fidelity and adaptation to reliably reproduce the intended outcomes.⁵⁶ Several common methods to guide and facilitate such assessments have been described.^27,51,57,58 Nevertheless, we know little about comprehensive plans or valid measures for measuring implementation and scaling-up fidelity.

Third, there is widespread enthusiasm for scaling up evidence-based innovations to improve primary care,^13,14 but evidence is limited on how to scale up effectively.^14,23,59 In Canada, the scaling up of innovations remains an understudied aspect of KT, shown by the fact that the widely used Knowledge-to-Action Framework does not include a scaling-up phase.⁵⁹ A systematic review showed that barriers to scaling up innovations in primary care include inconsistent reporting of data (e.g., no information on assessed scalability) and an absence of patient and public involvement.¹⁴ Also, sex (as a biological attribute) and gender (as a social construct) considerations have implications for scaling up in primary care.^20,60 User-informed assessments of innovations are essential parts of a scalability assessment.^8,24 To the best of our knowledge, no scalability assessment tools have rigorously integrated the reporting of assessment results or validation of data by users; these gaps will be addressed in the next version of our questionnaire.^20,36

Limitations

First, our study shares the general limitations of any approach using self-administered questionnaires (e.g., overestimation of evaluated criteria and reduction of objectivity) and cross-sectional studies (e.g., selection bias). One member of each team completed the survey, and there was no secondary objective assessment, nor do we know if this respondent consulted other team members. However, we had a satisfactory response rate.

Second, our survey collected data mostly on the presence of information necessary for assessing the scalability of innovations, rather than on the results of these assessments. However, our goal was to encourage innovation teams to improve the scalability of their projects and to target components requiring action. Future analyses should consistently collect the results of scalability assessments, as well as seek secondary, external and double-blinded evaluations to support evidence for these scores.

Finally, in our analysis, each criterion contributed equally to the scalability assessment score, while some criteria may be more relevant for some innovations than for others.

Conclusion

We explored scalability assessments of primary care innovations in the province of Quebec. These assessments varied according to the type of innovation. Management innovations integrated the most scalability components and may therefore be the type of innovation most prepared for primary care scale-up in Canada. Our findings contribute to important understanding of the strengths and weaknesses of scalability assessments in primary care innovation. Future surveys should include secondary validation of the assessment of scalability components and seek detailed results of these assessments.