Testing e-mail content to encourage physicians to access an audit and feedback tool: a factorial randomized experiment

G Vaisson; HO Witteman; S Chipenda-Dansokho; M Saragosa; Z Bouck; CA Bravo; L Desveaux; D Llovet; J Presseau; M Taljaard; S Umar; JM Grimshaw; J Tinmouth; NM Ivers

doi:10.3747/co.26.4829

. 2019 Jun 1;26(3):205–216. doi: 10.3747/co.26.4829

Testing e-mail content to encourage physicians to access an audit and feedback tool: a factorial randomized experiment

G Vaisson ^*,^✉, HO Witteman ^*, S Chipenda-Dansokho ^*, M Saragosa ^†, Z Bouck ^†, CA Bravo ^†, L Desveaux ^†, D Llovet ^†, J Presseau ^†, M Taljaard ^†, S Umar ^†, JM Grimshaw ^†, J Tinmouth ^†, NM Ivers ^†

PMCID: PMC6588071 PMID: 31285667

Abstract

Background

In Ontario, an online audit and feedback tool that provides primary care physicians with detailed information about patients who are overdue for cancer screening is underused. In the present study, we aimed to examine the effect of messages operationalizing 3 behaviour change techniques on access to the audit and feedback tool and on cancer screening rates.

Methods

During May–September 2017, a pragmatic 2×2×2 factorial experiment tested 3 behaviour change techniques: anticipated regret, material incentive, and problem-solving. Outcomes were assessed using routinely collected administrative data. A qualitative process evaluation explored how and why the e-mail messages did or did not support Screening Activity Report access.

Results

Of 5449 primary care physicians randomly allocated to 1 of 8 e-mail messages, fewer than half opened the messages and fewer than 1 in 10 clicked through the messages. Messages with problem-solving content were associated with a 12.9% relative reduction in access to the tool (risk ratio: 0.871; 95% confidence interval: 0.791 to 0.958; p = 0.005), but a 0.3% increase in cervical cancer screening (rate ratio: 1.003; 95% confidence interval: 1.001 to 1.006; p = 0.003). If true, that association would represent 7568 more patients being screened. No other significant effects were observed.

Conclusions

For audit and feedback to work, recipients must engage with the data; for e-mail messages to prompt activity, recipients must open and review the message content. This large factorial experiment demonstrated that small changes in the content of such e-mail messages might influence clinical behaviour. Future research should focus on strategies to make cancer screening more user-centred.

Keywords: Cancer screening, primary health care, audit and feedback, e-mail, persuasive communication, behaviour change techniques, factorial experiments, process evaluations

INTRODUCTION

Despite guidelines, many patients do not undergo recommended screening tests such as mammography (breast cancer), Pap tests (cervical cancer), and colonoscopy (colorectal cancer)¹^,². For example, in Ontario during 2012–2014, 35% and 37% respectively of the eligible population did not receive recommended mammography or Pap test screening, and 40% were overdue for colorectal cancer screening³. Cancer screening uptake depends on multiple factors at the patient, provider, and organization levels⁴^–⁷.

Cancer Care Ontario is the Ontario provincial agency responsible for organizing population-wide screening programs for breast, cervical, and colorectal cancers⁸. In many cases, eligible Ontarians must access those screening programs through a primary care provider. Recommendations, communication, and quality of the discussion between health care providers and their patients about cancer screening options are important determinants of the use of screening services⁶. Healthy People 2020, launched by the U.S. Department of Health and Human Services, set “increasing the proportion of adults who were counseled about cancer screening consistent with current guidelines” as one of its objectives⁹. A number of strategies, including electronic audit and feedback, have been attempted to encourage health professionals to change their behaviour so as to align their practice with evidence-based health directives, but so far, no optimal strategy has been identified for changing primary physician behaviour with respect to cancer screening¹⁰^–¹³.

Audit and feedback tools provide health professionals with a summary of their practice performance over a given period of time, comparing their performance with the performance of other professionals or with provincial or national standards¹⁰. The effects of audit and feedback vary depending on the way in which the intervention is designed and delivered¹⁴^,¹⁵. In fact, audit and feedback interventions can be considered a platform for delivering behaviour change techniques to support the delivery of guideline-concordant care¹⁶^,¹⁷. The Screening Activity Report created and administered by Cancer Care Ontario is an online audit and feedback tool that is updated monthly. Eligible primary care physicians in Ontario can register to receive a Screening Activity Report that lists their patients who are overdue for screening or who have received an abnormal result and require follow-up. Registered physicians (8462 at 1 May 2017) must work in a patient enrolment model—that is, having a known roster of patients for whom they are paid in part through capitation¹⁸. Registered physicians receive monthly e-mail messages from Cancer Care Ontario informing them that their report has been updated. Use of the Screening Activity Report is associated with slightly higher rates of cancer screening¹⁹, but only a small proportion of registered family physicians (38% in 2014) access their Screening Activity Report¹⁹.

In the present study, we aimed to examine the impact on Screening Activity Report access and cancer screening rates of 3 behaviour change techniques—anticipated regret, material incentive (behaviour), and problem-solving²⁰— within the monthly e-mail messages sent by Cancer Care Ontario to primary care physicians. We hypothesized that the behaviour change techniques would increase Screening Activity Report access and cancer screening rates.

METHODS

This pragmatic randomized factorial experiment and process evaluation used the Multiphase Optimization Strategy framework to optimize and evaluate multicomponent behavioural interventions²¹^,²². The study relates to the Preparation and Optimization phases of the framework. Details of the methods and intervention development have previously been described²³^,²⁴. An assessment of this pragmatic experiment scored 5 on a 5-point Likert scale for 7 of the 9 domains on the precis-2 tool²⁵ (supplemental Table 1).

Experimental Design, Setting, and Eligibility Criteria

Participants in this 2×2×2 randomized factorial experiment were primary care physicians (5449 in May 2017) who had accepted to receive routine e-mail messages about the Screening Activity Report. Nominated delegates (that is, people designated to access the Screening Activity Report on behalf of eligible physicians) were excluded, a decision that was made because of uncertainties in attributing the intervention to Screening Activity Report access by a delegate (that is, risk of contamination). The Research Ethics Board at Women’s College Hospital approved the study, with a waiver of informed consent for physician participation, given that the study met all Article 3.7A criteria of the Tri-Council Policy Statement²⁶^,²⁷. The study is registered at http://ClinicalTrials.gov/ as NCT03124316.

Interventions

In alignment with the Preparation phase of the Multiphase Optimization Strategy framework, the research team (including specialists in behaviour change theories and 2 senior research experts in qualitative methods) selected and drafted e-mail content based on literature about physician behaviour change, informed by the behaviour change techniques taxonomy (version 1) set out by Michie and colleagues²⁰^,²³. The first drafts of the e-mail message incorporated 6 behaviour change techniques (“active components of an intervention designed to change behavior”²⁸): anticipated regret, information about others’ approval, material incentive (behaviour), problem-solving, salience of consequences, and credible source.

To refine the intervention content, we then conducted two 2-hour workshops and three 2-hour focus groups with Screening Activity Report adopters and non-adopters, taking a user-centred approach²³. User-centred design is a highly iterative development process involving users or potential users early and often during the process to meet audience needs²⁹. It aims to develop useful products or tools that are adapted to people rather than to make people use a pre-specified product or tool that might not suit them²⁹.

During the first workshop, we asked the Screening Activity Report adopters to discuss their current practice for monitoring screening participation and talked about the benefits of the Screening Activity Report. We then asked them to write a letter to their colleagues that would convince those colleagues to use the Screening Activity Report. Finally, we showed them the pre-crafted messages containing the behaviour change techniques to get their initial reactions. Between workshops 1 and 2, we used their content and considered their reactions in developing an e-mail message that would be tested in workshop 2. During workshop 2, we received reactions to 2 e-mail messages with varied content. Between workshop 2 and the focus groups, we refined content. Finally, during the focus groups, we tested 2 e-mail messages with varied content and refined the content between the focus group sessions.

The final e-mail message operationalized 3 behaviour change techniques targeting different behavioural determinants²⁸. Those techniques were anticipated regret (targeting beliefs about consequences, intention, and emotions: “How would you feel if a patient had a poor outcome because you missed an abnormal test result?”), material incentive (behaviour) (targeting reinforcement: “Logging into the [Screening Activity Report] can help you maximize your screening rates and save time when calculating your preventive care bonus.”), and problem-solving (targeting behavioural regulation and environmental context and resources, including “Email ONE ID [College of Physicians and Surgeons of Ontario, Toronto, ON] to register a delegate with eHealth Ontario so they can check your report”)²⁸.

Testing the 3 behaviour change techniques in our factorial design resulted in 8 different versions of the e-mail message according to whether the message contained each of the 3 operationalized techniques [hereinafter called “factors” (supplemental Appendix 1): with no factors; with anticipated regret, or material incentive (behaviour), or problem-solving; with anticipated regret plus material incentive (behaviour); with anticipated regret plus problem-solving; with material incentive (behaviour) plus problem-solving; or with all 3 factors.

The e-mail messages were sent monthly by Cancer Care Ontario for 4 months between 10 May and 10 September 2017. Physicians received the same e-mail version every month.

Baseline Characteristics

Physician sex and years of practice [Corporate Provider Database (ices, Toronto, ON)], history of Screening Activity Report use [previous use or no previous use between October 2014 and April 2017, Reports Accessed by User Report (Cancer Care Ontario, Toronto, ON)], screening rates for breast, cervical, and colon cancer as of March 2017 [Regional Primary Care Provider Report (Health Quality Ontario, Toronto, ON)], and general practice characteristics [practice size, size of group, and rurality, Client Agency Program Enrolment (ices)] were obtained from routinely collected administrative data.

Outcomes

The primary outcome was Screening Activity Report access, defined as at least 1 log-in event during the 4-month experiment. Secondary outcomes were the number of different days in the 4-month period that participants logged in and the proportion of a physician’s enrolled eligible patients who were up-to-date for breast, cervical, and colorectal cancer screening at the end of the 4-month period. Data were not available for physicians who had fewer than 6 enrolled patients eligible for screening (data censored). We also tracked calls to Cancer Care Ontario and eHealth Ontario about the Screening Activity Report. The “open rate” of the e-mail message and the “click rate” (defined as the percentage of e-mail messages that attracted at least 1 click-through to the Screening Activity Report directly from the link in the message were assessed to evaluate intervention fidelity.

Data Collection

Routinely collected administrative data (ices) were used to obtain instances of Screening Activity Report access; up-to-date cancer screening status for breast, cervical, and colorectal cancer screening; baseline physician characteristics; and balance measures. Open and click rates were automatically collected by Mailchimp (https://mailchimp.com/ Rocket Science Group, Atlanta, GA, U.S.A.), the software automation platform used to send e-mail messages.

Power

The sample size was the number of eligible physicians who were registered and appeared on the active user list of the Screening Activity Report at the time of randomization. After removal of duplicate e-mail addresses, the total number of unique eligible physicians was 5449. That sample size achieves a 94% power to detect an absolute difference of 4% in Screening Activity Report use (between participants with the factor present and those with the factor absent) using a 2-sided test at the 5% level of significance, assuming a control arm proportion of 0.20.

Randomization, Allocation Concealment, and Blinding

The allocation schedule was computer-generated by the study biostatistician (ZB) using simple unrestricted randomization. The allocation sequence was applied to a de-identified list of eligible physicians (May 2017) exported from Cancer Care Ontario. Because Cancer Care Ontario could distribute only one type of e-mail message at a time, the order in which the 8 message types were sent was determined randomly every month. The e-mail message versions were sent at 45-minute intervals starting at 09h00 on the 10th day of each month (or if that day fell on a weekend, on the next business day) between 10 May and 10 September 2017.

Randomization was conducted using the SAS software application (version 9.4: SAS Institute, Cary, NC, U.S.A.).

Statistical Analysis

The primary outcome was analyzed using robust Poisson regression analysis³⁰ rather than the logistic regression analysis originally planned²⁴, yielding estimates as relative risks (rrs) rather than as odds ratios. The secondary outcomes (number of times the Screening Activity Report was accessed and cancer screening rates) were analyzed using a log-Poisson regression yielding estimated rate ratios (rars). The unit of analysis was the individual physician. The 3 intervention components were entered as main effects using effects coding to express the marginal effect of each intervention component; that is, the estimated rr or rar represents the average effect across conditions with the particular intervention component present (for example, the average effect of anticipated regret on primary care physicians allocated to study arms 2 + 5 + 6 + 8) compared with conditions in which the intervention component was absent (the average effect of anticipated regret on primary care physicians allocated to study arms 1 + 3 + 4 + 7). Secondary analyses examined 2-way interactions between the intervention components. Per the pre-specified protocol, the primary analysis was adjusted for history of Screening Activity Report use. Pre-specified subgroup analyses examined effect modification through stratification by physician history of report access (binary) and sex (men vs. women). We had planned to explore additional effect modifiers, but given the absence of significant findings for the main effects, we chose to forego such analyses to avoid unwarranted increases in the risk of type i error. Payment models could not be tested because, counter to our original analytic plan²⁴, data were not available from Cancer Care Ontario.

The analyst (GV) was not blind to experimental arms. The threshold for statistical significance was 0.05, and all analyses were conducted in the SAS software application.

Process Evaluation

We explored how and why the e-mail messages did or did not prompt physicians to access the Screening Activity Report.

Primary care physicians were recruited by convenience sampling, through the personal network of the principal investigator (NMI) and through snowball sampling as a secondary strategy. Because of privacy regulations, specific contact with participating physicians in each study arm was not possible. Physicians were eligible if they were randomized to receive an intervention e-mail message as part of the study. Between August and September 2017, consenting physicians participated in a 30-minute semi-structured telephone interview and received an honorarium of CA$150 in appreciation of their time. Physicians were recruited until data saturation was reached—that is, when no new comments were heard during the interviews³¹.

A first interview guide was developed by GV, MS, LD, NMI, and HOW²⁴ and was later adapted into 8 versions to match the e-mail versions received by the physicians (supplemental Appendix 2). Interview guide development was led by the Theoretical Domains Framework. Questions therefore sought to prompt physicians about the determinants of behaviour relating to Screening Activity Report access and cancer screening guideline adherence. If physicians could not retrieve the version of the e-mail message they had received, the version containing the anticipated regret, material incentive, and problem-solving factors was sent before the interview. All interviews were audio-recorded and transcribed.

The initial deductive framework was derived from the 3 behaviour change techniques²⁰. However, because most physicians had not opened the intervention e-mail messages and were therefore not exposed to the intervention, we used framework analysis³² to develop summary categories that captured the central themes of the interviews. The lead author (GV), together with an experienced mixed-methods researcher (SCD), generated a set of initial themes after coding 3 interviews. Then, GV and SCD refined and sought themes in an iterative manner until no new themes were identified. Supplemental Table 2 presents the final codes. Using the new coding structure, all data were coded by 2 independent analysts (GV, SCD) using the NVivo software application (version 11: QSR International, Melbourne, Australia).

RESULTS

Participant Flow

The study included 5449 physicians, 2606 of whom received e-mail messages targeting anticipated regret; 2744, material incentive; and 2776, problem-solving. The study flow diagram (Figure 1) details the number of physicians receiving each e-mail version. Each month, between 4.6% and 7.3% of the messages bounced back (supplemental Table 3). The final retention rate was 99.0% (5396/5449, Figure 1).

Baseline Characteristics

Baseline characteristics of the participants were similar for all 3 factors (Table I) and the 8 different e-mail versions (supplemental Table 4).

TABLE I.

Baseline physician characteristics according to each operationalized behaviour change technique

Characteristic	Anticipated regret	Material incentive (behaviour)	Problem-solving
Physicians (n)	2606	2744	2776

History of SAR use^a
Median	0.03	0.03	0.03
IQR	0–0.17	0–0.13	0–0.13

Baseline cancer screening rate^b (%)
Breast
Median	0.64	0.65	0.65
IQR	0.57–0.72 (15 missing)	0.57–0.72 (13 missing)	0.57–0.71 (14 missing)
Cervical
Median	0.67	0.67	0.67
IQR	0.58–0.75 (13 missing)	0.57–0.74 (10 missing)	0.57–0.74 (10 missing)
Colon
Median	0.69	0.69	0.69
IQR	0.62–0.75 (7 missing)	0.61–0.75 (5 missing)	0.62–0.75 (4 missing)

Sex [n (%)]
Women	1246 (47.8)	1333 (48.6)	1326 (47.8)
Men	1360 (52.2)	1411 (51.4)	1450 (52.2)

Years in practice^c
Median	8.1	8.2	8.2
IQR	4.9–10.6	5.1–10.7	5.2–10.7

Practice size^d
Median	664	670	668
IQR	481–896 (13 missing)	488–895 (22 missing)	485–895 (19 missing)

Size of group^e
Median	16	16	15
IQR	8–32 (1 missing)	8–32 (7 missing)	8–31 (8 missing)

Rurality [n (%)]
Rural	215 (8.3)	228 (8.3)	225 (8.1)
Urban	2374 (91.1)	2490 (90.7)	2529 (91.1)
Missing	17 (0.65)	26 (0.95)	22 (0.79)

Open in a new tab

Monthly average SAR access between 23 October 2014 and 30 April 2017, considering the number of months each participant was registered to use the SAR and to receive the e-mail message from Cancer Care Ontario.

At March 2017, pre-trial.

From year of graduation.

Number of patients enrolled to the physician’s practice.

Total number of physicians belonging to the group and practicing within the Local Health Integration Network.

SAR = Screening Activity Report; IQR = interquartile range.

Screening Activity Report Access by Factor

Table II presents descriptive summaries of the primary and secondary outcomes. On average, physicians accessed their Screening Activity Report less than 1 time during the 4 months of follow-up regardless of factor or e-mail version (Table II and supplemental Table 5). The Screening Activity Report was accessed by 567 (21.8%), 558 (20.3%), and 540 (19.5%) of the physicians in the anticipated regret, material incentive, and problem-solving groups respectively. The primary outcome is presented in supplemental Table 5 by e-mail version. The e-mail messages containing only the material incentive factor had the highest access rate (24.2%); e-mail messages containing both the material incentive and problem-solving factors had the lowest rate (17.3%) (supplemental Table 5).

TABLE II.

Descriptive statistics for outcomes according to each operationalized behaviour change technique

Outcome	Anticipated regret (n=2606)	Material incentive (behaviour) (n=2744)	Problem-solving (n=2776)
Primary
SAR access [n (%)]
Yes	567 (21.8)	558 (20.3)	540 (19.5)
No	2039 (78.2)	2186 (79.7)	2236 (80.6)

Secondary
Mean times SAR was accessed	0.60±2.25	0.50±1.70	0.48±1.87
Breast cancer screening rate^a (%)
Median	0.65	0.64	0.65
IQR	0.57–0.71 (57 missing)	0.57–0.71 (71 missing)	0.57–0.71 (68 missing)
Cervical cancer screening rate (%)
Median	0.67	0.67	0.67
IQR	0.58–0.74 (57 missing)	0.57–0.74 (69 missing)	0.58–0.75 (63 missing)
Colon cancer screening rate (%)
Median	0.70	0.69	0.69
IQR	0.63–0.76 (50 missing)	0.62–0.76 (63 missing)	0.62–0.76 (58 missing)

Process
Mean calls to Cancer Care Ontario	1.2±0.5 (n=17)^b	1.2±0.6 (n=13)	1.2±0.5 (n=24)
Mean calls to eHealth Ontario	1.0±0 (n=4)	1 (n=1)	1.0±0 (n=6)
Participants dropping out^c [n (%)]	21 (0.8)	23 (0.8)	23 (0.8)
Participants unsubscribing^d [n (%)]	11 (0.4)	9 (0.3)	6 (0.2)
Participants dropping out or unsubscribing [n (%)]	27 (1.0)	30 (1.1)	27 (1.0)

Open in a new tab

At September 2017, post-trial.

Total number of physicians calling Cancer Care Ontario or e-Health about the SAR.

SAR account no longer active (physicians did not access the SAR at least once in the preceding year).

Physicians no longer receiving the e-mail message from Cancer Care Ontario.

SAR = Screening Activity Report; IQR = interquartile range.

Main Effects on Screening Activity Report Access

Table III presents the results from the multivariable Poisson regression analysis of the primary outcome. No factor was associated with a statistically significant increase in Screening Activity Report use. However, after adjustment for history of Screening Activity Report use, problem-solving e-mail messages were associated with a statistically significant 12.9% relative reduction in Screening Activity Report access [rr: 0.871; 95% confidence interval (ci): 0.791 to 0.958; p = 0.005]. The material incentive factor (rr: 0.954; 95% ci: 0.867 to 1.050; p = 0.34) was also associated with a reduction (but not a statistically significant one), and the anticipated regret factor was associated with a nonsignificant increase (rr: 1.072; 95% ci: 0.975 to 1.180; p = 0.15).

TABLE III.

Multivariable Poisson regression analysis of primary and secondary outcomes

Outcome	Comparison	Unadjusted analysis			Adjusted analysis

		RR	95% CI	p Value	RR	95% CI	p Value
Primary (SAR access)

Anticipated regret	Present vs. absent	1.096	0.988 to 1.216	0.08	1.072	0.975 to 1.180	0.15
Material incentive	Present vs. absent	0.959	0.864 to 1.064	0.43	0.954	0.867 to 1.050	0.34
Problem-solving	Present vs. absent	0.879	0.792 to 0.990	0.02	0.871	0.791 to 0.958	0.005
History of SAR use^a					8.4	6.9 to 10.2	<0.0001

Secondary

Times the SAR was accessed
Anticipated regret	Present vs. absent	1.127	0.963 to 1.319	0.14	1.090	0.950 to 1.251	0.22
Material incentive	Present vs. absent	0.902	0.771 to 1.056	0.20	0.887	0.773 to 1.018	0.09
Problem-solving	Present vs. absent	0.832	0.711 to 0.974	0.02	0.813	0.708 to 0.934	0.003
History of SAR use					12.6	9.5 to 16.6	<0.0001

Breast cancer screening
Anticipated regret	Present vs. absent	1.004	0.995 to 1.013	0.43	1.000	0.997 to 1.002	0.81
Material incentive	Present vs. absent	1.001	0.992 to 1.010	0.90	1	0.997 to 1.003	0.94
Problem-solving	Present vs. absent	0.998	0.989 to 1.1.007	0.69	0.998	0.995 to 1.001	0.13
Baseline breast cancer screening^b					6.9	6.8 to 7.1	<0.0001

Cervical cancer screening
Anticipated regret	Present vs. absent	1.001	0.992 to 1.011	0.83	1.002	0.998 to 1.003	0.83
Material incentive	Present vs. absent	1.006	0.996 to 1.015	0.24	1.000	0.998 to 1.002	0.81
Problem-solving	Present vs. absent	0.999	0.989 to 1.008	0.77	1.003	1.001 to 1.006	0.003
Baseline cervical cancer screening^b					7.1	7.0 to 7.2	<0.0001

Colorectal cancer screening
Anticipated regret	Present vs. absent	0.995	0.987 to 1.003	0.21	1	0.998 to 1.002	0.99
Material incentive	Present vs. absent	1.000	0.992 to 1.008	0.95	1	0.998 to 1.002	0.93
Problem-solving	Present vs. absent	0.996	0.988 to 1.004	0.33	1	0.998 to 1.002	0.95
Baseline colorectal cancer screening^b					6.8	6.7 to 6.9	<0.0001

Open in a new tab

Monthly average SAR access between 23 October 2014 and 30 April 2017 considering the number of months that each participant was registered to use the SAR and was receiving the e-mail message from Cancer Care Ontario.

At March 2017, pre-trial.

RR = risk ratio; CI = confidence interval; SAR = Screening Activity Report.

Main Effects on the Number of Times the Screening Activity Report Was Accessed

Interpretation of results for the number of times that the Screening Activity Report was accessed (Table III) is similar to that for the primary outcome. After adjustment for history of Screening Activity Report use, the problemsolving factor was associated with a statistically significant decrease of 18.7% in the number of times the Screening Activity Report was accessed (rar: 0.813; 95% ci: 0.708 to 0.934; p = 0.003). The material incentive factor was also associated with a decrease, but not a statistically significant one (rar: 0.887; 95% ci: 0.773 to 1.018; p = 0.09). The association with anticipated regret was positive, but not statistically significant (rar: 1.090; 95% ci: 0.950 to 1.251; p = 0.22).

E-Mail Message Open and Click Rates

In May, open rates varied from 50% to 57%, and click rates varied from 7% to 14% (supplemental Table 3). Open rates then steadily declined to 45%–55% in June, to 44%–47% in July, and to 40%–46% in August. Click rates decreased to 6%–11% in June, to 7%–10% in July, and to 4%–8% in August (supplemental Table 3).

Main Effects on Up-to-Date Breast, Cervical, and Colorectal Cancer Screening Status

No e-mail message was associated with a statistically significant effect on rates of breast cancer screening or colorectal cancer screening. But after adjusting for the cervical cancer screening baseline, a statistically significant positive effect on cervical cancer screening was observed for problem-solving e-mail messages (rar: 1.003; 95% ci: 1.001 to 1.006; p = 0.003; Table III). With 2,365,062 patients being eligible for cervical screening, that difference represents 7568 more patients being screened during the 4 months.

Ancillary Analyses

Exploratory Interactions and Modification Effects

Table IV presents results from the exploratory subgroup analyses stratified by sex and history of Screening Access Report use. The pattern of results was observed to be similar for the male and female physicians. In particular, for male and female physicians alike, Screening Activity Reports were accessed less frequently by those receiving the problemsolving e-mail messages than by those not receiving those messages (rr for men: 0.849; 95% ci: 0.740 to 0.975; p = 0.02; rr for women: 0.895; 95% ci: 0.783 to 1.022; p = 0.10).

TABLE IV.

Factors affecting the primary outcome, stratified by sex and history of Screening Activity Report (SAR) use

Primary outcome (SAR access)	Comparison	Unadjusted analysis			Adjusted analysis

		RR	95% CI	p Value	RR	95% CI	p Value
Women (n=2607)
Anticipated regret	Present vs. absent	1.040	0.902 to 1.200	0.59	1.023	0.896 to 1.169	0.73
Material incentive	Present vs. absent	0.905	0.784 to 1.044	0.17	0.918	0.804 to 1.049	0.21
Problem-solving	Present vs. absent	0.888	0.769 to 1.024	0.10	0.895	0.783 to 1.022	0.10
History of SAR use^a					7.2	5.5 to 9.3	<0.0001

Men (n=2842)
Anticipated regret	Present vs. absent	1.158	0.996 to 1.346	0.06	1.124	0.980 to 1.290	0.09
Material incentive	Present vs. absent	1.0156	0.874 to 1.171	0.84	0.991	0.8633 to 1.137	0.89
Problem-solving	Present vs. absent	0.873	0.751 to 1.014	0.08	0.849	0.740 to 0.975	0.02
History of SAR use					9.7	7.4 to 12.8	<0.0001

No history of SAR use (n=2581)
Anticipated regret	Present vs. absent	0.850	0.586 to 1.234	0.39	0.850	0.586 to 1.234	0.39
Material incentive	Present vs. absent	1.054	0.728 to 1.526	0.78	1.054	0.728 to 1.526	0.78
Problem-solving	Present vs. absent	0.963	0.665 to 1.394	0.84	0.963	0.665 to 1.394	0.84

History of SAR use (n=2868)
Anticipated regret	Present vs. absent	1.098	0.995 to 1.212	0.06	1.098	0.995 to 1.212	0.06
Material incentive	Present vs. absent	0.944	0.856 to 1.042	0.25	0.944	0.856 to 1.042	0.25
Problem solving	Present vs. absent	0.862	0.781 to 0.951	0.003	0.862	0.781 to 0.951	0.003
History of SAR use

Open in a new tab

Monthly average SAR access between 23 October 2014 and 30 April 2017, considering the number of months each participant was registered to use the SAR and receive the e-mail message from Cancer Care Ontario.

RR = risk ratio; CI = confidence interval.

Subgroup differences were observed for physicians with and without a history of access to Screening Activity Reports. For those previously without access, the e-mail messages had no association with statistically significant effects; however, for physicians with previous access to the Screening Activity Report (compared with their counterparts not having access), the receipt of problem-solving e-mail messages was associated with a statistically significant 13.8% lower risk of Screening Activity Report access (rr: 0.862; 95% ci: 0.781 to 0.951; p = 0.003; Table IV).

Analyses examining two-way interactions between the factors found no statistically significant interactions (supplemental Tables 6 and 7).

Process Outcomes by Factor

The problem-solving factor was associated with the number of calls to Cancer Care Ontario or eHealth (rar: 2.226; 95% ci: 1.055 to 4.694; p = 0.04), but anticipated regret and material incentive were not (anticipated regret rar: 1.164; 95% ci: 0.586 to 2.312; p = 0.67; material incentive rar: 0.560; 95% ci: 0.274 to 1.142; p = 0.11; supplemental Table 8).

Drop-outs and unsubscribing were not associated with any factor (anticipated regret rr: 1.130; 95% ci: 0.661 to 1.933; p = 0.65; material incentive rr: 1.284; 95% ci: 0.748 to 2.206; p = 0.36; problem-solving rr: 0.997; 95% ci: 0.583 to 1.704; p = 0.99; supplemental Table 8).

Process Evaluation

Supplemental Table 9 describes the characteristics of the physicians interviewed (n = 11). All physicians used electronic medical records, with a median of 7 years of use (interquartile range: 5–8 years), and 55% accessed their Screening Activity Report during the study.

Physicians reported using the monthly e-mail messages from Cancer Care Ontario simply as a reminder to check their Screening Activity Report at a later time, regardless of the content of the message. Only 1 physician interviewed used the link in the message to access the Screening Activity Report. Most physicians reported that they opened and glanced at, but did not attentively read, the messages. The physicians who opened the messages with the anticipated regret content reported that it provoked an emotional reaction, but no follow-up action. The material incentive content led to a discussion of the value of incentives, and the problem-solving suggestions were perceived as interesting, but not actionable, even though the suggestion to delegate responsibility was appealing.

Physicians also offered a number of suggestions to improve the messages. They asked for a brief, personalized message including the specific results and peer comparisons, although such content raises privacy concerns²³. One also asked for the message to be sent to a delegate (for example, the clinic nurse), something suggested in the problem-solving messages; however, other respondents did not find that solution actionable either because their practice was new or because no colleague in the clinic was perceived to be worthy of that trust. Another suggestion was to send the e-mail messages with different information each time, possibly supplemented with links to guidelines, or even to send it by postal mail. Finally, physicians mentioned factors determining or facilitating their adherence to cancer screening guidelines. Such factors included knowing or learning more about using their electronic medical record to track screening, receiving incentives (financial or continuing medical education credits), and obtaining the results of tests ordered by other physicians (for example, gynecologists).

Regardless of whether they opened or did not open the e-mail messages, physicians typically had other systems in place (for example, electronic medical records) to track the cancer screening needs of their patients. Physicians who had other systems in place found that the Screening Activity Report was comparatively time-consuming, unhelpful, and redundant. The most commonly identified added value of the Screening Activity Report occurred when physicians took on new patients (and therefore had no patient history or record) or when their patients underwent tests ordered by other physicians. The Screening Activity Report was also useful in instances in which physicians had to validate electronic medical record data. Physicians made a number of suggestions to improve the Screening Activity Report, including integration of the report data into their electronic medical record; facilitating access to the report; and providing more accurate and “actionable” data (that is, personalized information that could be acted on immediately). Table V presents illustrative quotes from participants.

TABLE V.

Qualitative process evaluation (11 respondents)

Theme	Sample quotes
SAR monthly e-mail habits	So anytime I get a SAR [e-mail reminder], it goes straight in the unread inbox and I look at it later, whether later in the day, another day or a couple of days later or on the weekend. So the fact that I get it during the day or at night is irrelevant, because I never look at it right away. — Participant 10

Utility of the SAR	It doesn’t help me maximize my screening rates, and it certainly doesn’t help me save time—it adds time.... It’s not providing us any more useful information than what they already have in their EMR.... We’ve never used the SAR well, because we already had a process in place that worked before the SAR came into being. — Participant 1
Utility of the SAR	We’ll correlate three sources. So we’ll correlate the paper format that we get from OHIP and the call backs and the nurse’s updated scripts, I guess, that she, you know, handwrites on the OHIP. We correlate that with SARs, and then we correlate it as well with our EMR by doing a database search.... I do access it personally in October, usually October, November in order to anticipate following to try to capture as many as I can at that time.... I think I look at the SAR as a complement to what we’re doing. — Participant 3

SAR-to-EMR comparisons	Like I never actually went into the many details to see if the SAR is as accurate as I would hope it to be, but I know that our EMR is accurate because if it’s been done, and I have a result only then it’s documented.... So, actually, what I do is that I go to our EMR list of people who have done, had it done, and then I use the SAR to kind of reconcile. — Participant 2

Suggestions to improve the utility or use of the SAR	I mean just maybe two things. Like, one thing, we always educated them [about the SAR]. Like, our people are aware of it. I mean sending out an e-mail is one thing, but finding other ways to encourage people and to motivate them to use it. The other thing would be maybe finding other ways to integrate like with EMRs. You know, it’d be nice. Again, our EMR does spit out this data anyways, but again maybe to help reconcile those few patients. It’d be so nice if I didn’t have to manually do it. I think that would take away a huge workload. — Participant 2
Suggestions to improve the utility or use of the SAR	If it was quick, I could click one button and get in ... Abnormal screen, overdue for screening.... You can’t put individual patients’ information in there, but if you could put individual physician information. In other words, you have x number of patients requiring this or you have x number of patients. Like, just putting, like, that summary sheet at the front that tells us may be outstanding for our patients might be helpful. — Participant 4

Utility of the regular SAR e-mail messages	Receiving the e-mail? Well, it’s a prompt, so it does keep it in mind. So, if I never received an e-mail, I probably would never check my SAR, because it would just—I’d never think about it. — Participant 10
Utility of the regular SAR e-mail messages	I mean definitely when I first started, you know, I think the e-mails were super helpful because that’s how I learned about this resource, you know. But as I get a system in place, I kind of know now that, hey, you know, at this time of year, maybe twice a year at minimum, I’m going to go and just update our records. — Participant 2

Suggestions to improve the utility of the regular SAR e-mail messages	if I had to get the SAR e-mail every week, I’d be like, delete, delete, delete, and then on the one time when there was something important in it, I’d delete it without looking.... And depending on the urgency, I mean, once a month might be good for some things, and once every two months for some other things ... so that you aren’t inundated with too much information each time, that kind of thing. — Participant 10
	Well, if, you know, confidentially ... weren’t such an issue, I would love this stuff to be e-mailed to us in a chart form, you know. But obviously that’s not a possibility.... I think if the e-mail had some of my personal summaries in it ... that would much more likely get me to log in than just a general reminder. — Participant 4
	Just give me my results, I can open the attachment and look at it. Or give me a Web site where all I have to do is click on that thing and it automatically takes me into what I have to look at. I don’t have to log in, password, like all these steps are annoying, they take too much time. — Participant 8

Determinants to adhere to cancer screening guidelines	Money would be much better spent teaching physicians how to use their EMRs properly, instead of having a duplication of the data in another format, that is not integrated with the EMR. — Participant 1
Determinants to adhere to cancer screening guidelines	The financial motivation is definitely a factor that allows me to put some time aside out of the rest of my day to do that.... We all feel constantly drowning in tasks that need to get done, that we’re pulled in so many different directions in terms of patient care, in terms of teaching, in terms of administrative, that it’s just hard to find a chunk of time that is required to sit down and do this. And so any sort of incentive that says there’s an additional benefit to doing this is helpful, to be honest. — Participant 4

Open in a new tab

SAR = Screening Activity Report; EMR = electronic medical record; OHIP = Ontario Health Insurance Plan.

DISCUSSION

In this study, we used the Multiphase Optimization Strategy to examine the effect of component behaviour change techniques within a monthly e-mail message sent to primary care physicians encouraging them to access a cancer screening audit and feedback report. In an effort to inform ongoing initiatives, we were able to embed this pragmatic factorial experiment involving more than 5000 physicians within the routine operations of our health system partner³³. For audit and feedback to work, recipients must engage with the data³⁴; for an e-mail message to prompt activity, recipients must open and review the e-mail content.

During the study, only about half the e-mail messages were opened, and in any given month, fewer than 11% of recipients “clicked through” the message to the Screening Activity Report. Those e-mail open and click rates are well above the mean open rate (21%) and click rate (2%) reported by Mailchimp for e-mail messages in the medical, dental, and health care industries. Those overall modest rates might reflect the challenges of using e-mail as a method to reach busy health professionals. Our qualitative findings suggest that physicians might not read the Screening Activity Report e-mail message and that some leave it flagged in their inbox to deal with later. Coupled with the qualitative findings, the low open rates suggest that the program’s theory must be revisited³⁵. The results suggest that the choice to open, read, and act on a reminder e-mail message prompting a physician to check the Screening Activity Report depends on the importance or pertinence that the physician gives to the Screening Activity Report in relation to other priorities and in relation to other strategies for achieving the goal of cancer screening.

Operationalization of anticipated regret and material incentive within monthly e-mail messages to physicians did not influence Screening Activity Report access. Physicians receiving a message containing the problem-solving behaviour change technique were 13% less likely to access the Screening Activity Report during the 4-month study period. We operationalized problem-solving as a bulleted list: “Three tips from other Ontario family doctors on how to fit using the Screening Activity Report into your schedule.” The tips were to delegate access to a member of staff by sending a message to a provided e-mail address, booking time in their calendar to review the report, or “tackle a few patients at a time.” The effect of those problem-solving tips demonstrates that specific content choices in physician communication can influence behaviour.

We speculate that the specific response could reflect one of two effects. First, physicians who received the recommendation to authorize a delegate to access the report on their behalf might have followed that recommendation and thus reduced their own report access. The increase in cervical cancer screening with the problem-solving factor also supports that hypothesis, although inferences about the small observed effect in that secondary outcome are at risk of type i error, especially given the large number of statistical tests conducted. Second, problem-solving content might not be helpful when motivation to complete a task is not already high. In such cases, it is possible that emphasizing the difficulties of staying on top of reports fails to help recipients overcome gaps between intention and behaviour³⁶. Further data about delegates will be necessary before firm recommendations can be made.

In the participatory design process and in the qualitative interviews described here, physicians noted other changes at the system level that could make it easier for them to use the data in their Screening Activity Report. They emphasized a desire for such reports to be integrated into the electronic medical records that they already use or to receive actionable data directly by e-mail. Thus, the operationalization of problem-solving might not have adequately addressed key barriers to the desired action. Those findings suggest that individuals developing audit and feedback interventions should carefully consider how to align their intervention with the workflow of the people who will use it. When the goals of the organization and of the users do not align, the ability to bring about change could be limited, even when using best practices in design.

Our study had 3 methodologic strengths. First, we were able to test, at scale, interventions that had been carefully co-designed with users, in partnership with a provincial agency. Second, rates of bounce-back, loss to follow-up, and unsubscribing were low (<10%), meaning that selection bias was negligible. Third, we conducted qualitative interviews to help explain the quantitative results. Those interviews provided further insight into how to optimize cancer screening in primary care.

Limitations of the study included, first, a temporal window for observing changes that might not have been ideal. Physicians reported consulting their Screening Activity Report typically before receiving preventive care bonuses in fall and winter. Summer might also mean fewer medical visits. It was also not possible to restrict analyses to patients who visited their physician during the 4 months of the study. Second, outcome measures were automatically generated and collected from Mailchimp and Cancer Care Ontario databases. That approach avoided any overestimation of cancer screening rates that might have occurred if physicians self-reported³⁷, which could have led to non-differential information bias and sub-estimation of effects. However, Mailchimp data could not be merged with Cancer Care Ontario data, meaning that measuring effect for physicians who opened their e-mail messages was not possible. Third, randomization was done at the physician level and not the practice level, meaning that contamination is plausible. However, the risk of cross-exposure seems low, given the low open rates. Fourth, for the process evaluation, we used a convenience sample recruited through the principal investigator of the study. That sampling technique could have affected the diversity of the sample. Five, we selected and operationalized behaviour change techniques that seemed most likely to be effective, but it is plausible that, despite our best efforts, we selected the wrong techniques or operationalized them inadequately. Finally, we did not test different e-mail subject lines for the various experimental conditions or over time, which might have contributed to declining open rates. Future studies of e-mail prompts to physicians should consider randomizing and testing a variety of subject lines.

The findings about effective communication techniques to prompt action by primary care physicians are broadly applicable to other physicians in Canada and abroad. Qualitative insights from our process evaluation indicate that Screening Activity Report use depends on multiple practical factors that limit its fit within existing workflows. As in many evaluations of implementations of e-health tools specifically (or practice-change initiatives more broadly), the innovation must align with intra- and inter-organizational structures involved in a specific care process³⁸^–⁴¹. Before implementing any new tool related to cancer screening (or any other process) into practice, it is necessary to determine aspects of the existing local processes where capacity and desire for change are both present. Foundational work to illuminate the realities “on the ground”—for example, surveying, meeting, interviewing, or shadowing the target population during a typical day of practice—should inform implementation strategies to improve quality of care.

Reflections on the Value of Catalyst Project Funding

This pragmatic experiment, conducted in collaboration with Cancer Care Ontario, allowed for a largescale evaluation of multiple interventions. The catalyst project was an ideal circumstance in which to make use of the first steps in the Multiphase Optimization Strategy framework for optimizing multicomponent interventions. The partnered approach also enabled the team to understand the constraints of real-world implementation and to inform future work. The grant provided an opportunity to apply principles of behavioural science to try to improve cancer care. Thus, in addition to the scientific findings, we highlight the benefits gained through working with a health care organization and conducting a trial using real-world data.

CONCLUSIONS

In this large factorial experiment, we tested 3 behaviour change techniques embedded in e-mail content sent to more than 5000 primary care physicians to increase the use of an audit and feedback tool for cancer screening. Despite limited engagement by the participants with the intervention, we found that small changes in the content of e-mail communications to physicians can lead to behaviour change, with the potential for change in clinical outcomes. The study demonstrated the potential benefits of rigorous partnered research with health system organizations to examine ways to optimize quality improvement interventions. This type of large-scale implementation research is crucial when working on population-level interventions in which very small effects could be worthwhile. The study also highlights that improving uptake of audit and feedback is a complex problem, unlikely to be satisfactorily addressed with single or simple interventions. Future work in the area should focus on the key goal of increasing cancer screening rather than increasing use of a particular tool, given that the principle of user-centred design is to make systems useful to people rather than to make people use specific systems.

Supplemental Materials

CONC-26-205-s001.pdf^{(1.1MB, pdf)}

ACKNOWLEDGMENTS

We thank all the participating physicians for their insights. Special thanks go to Petra de Heer from Cancer Care Ontario. This study was conducted with the support of the Ontario Institute for Cancer Research and Cancer Care Ontario through funding provided by the Government of Ontario. Funding for GV’s doctoral work was provided by that grant and by a Canadian Institutes of Health Research (cihr) Foundation Grant (fdn 148426, principal investigator HOW) during the preparation of the manuscript. HOW is supported by a Fonds de recherche du Québec–Santé Research Scholar Junior 2 salary award. NMI was supported by a cihr New Investigator Award and a Clinician Scientist Award from the Department of Family and Community Medicine at the University of Toronto and by the Women’s College Hospital Chair in Implementation Science. JMG holds a Canada Research Chair in Health Knowledge Transfer and Uptake and a cihr Foundation Grant (fdn 143269).

Footnotes

Supplemental material available at http://www.current-oncology.com

CONFLICT OF INTEREST DISCLOSURES

We have read and understood Current Oncology’s policy on disclosing conflicts of interest, and we declare the following interests: During the course of the project, authors CAB, DL, SU, and JT were employed at Cancer Care Ontario. The remaining authors have no conflicts to disclose.

REFERENCES

1.Kerner J, Liu J, Wang K, et al. Canadian cancer screening disparities: a recent historical perspective. Curr Oncol. 2015;22:156–63. doi: 10.3747/co.22.2539. [DOI] [PMC free article] [PubMed] [Google Scholar]
2.Anas R, Bell R, Brown A, Evans W, Sawka C. A ten-year history: the Cancer Quality Council of Ontario. Healthc Q. 2012;15(spec):24–7. doi: 10.12927/hcq.2012.23162. [DOI] [PubMed] [Google Scholar]
3.Cancer Care Ontario (cco) Ontario Cancer Screening Performance Report 2016. Toronto, ON: cco; 2016. [Google Scholar]
4.Wools A, Dapper EA, de Leeuw JRJ. Colorectal cancer screening participation: a systematic review. Eur J Public Health. 2016;26:158–68. doi: 10.1093/eurpub/ckv148. [DOI] [PubMed] [Google Scholar]
5.De Klerk CM, Gupta S, Dekker E, Essink-Bot ML on behalf of the Expert Working Group “Coalition to reduce inequities in colorectal cancer screening” of the World Endoscopy Organization. Socioeconomic and ethnic inequities within organised colorectal cancer screening programmes world-wide. Gut. 2018;67:679–87. doi: 10.1136/gutjnl-2016-313311. [DOI] [PubMed] [Google Scholar]
6.Peterson EB, Ostroff JS, DuHamel KN, et al. Impact of provider–patient communication on cancer screening adherence: a systematic review. Prev Med. 2016;93:96–105. doi: 10.1016/j.ypmed.2016.09.034. [DOI] [PMC free article] [PubMed] [Google Scholar]
7.Anhang Price R, Zapka J, Edwards H, Taplin SH. Organizational factors and the cancer screening process. J Natl Cancer Inst Monogr. 2010;2010:38–57. doi: 10.1093/jncimonographs/lgq008. [DOI] [PMC free article] [PubMed] [Google Scholar]
8.Cancer Care Ontario (cco) Who We Are [Web page] Toronto, ON: cco; 2015. [Available at: https://www.ccohealth.ca/en/who-we-are; cited 30 May 2017] [Google Scholar]
9.United States Department of Health and Human Services, Office of Disease Prevention and Health Promotion, Healthy People 2020. Cancer [Web page, Objectives tab] [Available at: https://www.healthypeople.gov/2020/topics-objectives/topic/cancer/objectives; cited 3 April 2019]
10.Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback : effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012:CD000259. doi: 10.1002/14651858.CD000259.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
11.Scott A, Sivey P, Ait Ouakrim D, et al. The effect of financial incentives on the quality of health care provided by primary care physicians. Cochrane Database Syst Rev. 2011:CD008451. doi: 10.1002/14651858.CD008451.pub2. [DOI] [PubMed] [Google Scholar]
12.Arditi C, Rège-Walther M, Wyatt JC, Durieux P, Burnand B. Computer-generated reminders delivered on paper to healthcare professionals; effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2012;12:CD001175. doi: 10.1002/14651858.CD001175.pub3. [DOI] [PubMed] [Google Scholar]
13.Tuti T, Nzinga J, Njoroge M, et al. A systematic review of electronic audit and feedback: intervention effectiveness and use of behaviour change theory. Implement Sci. 2017;12:61. doi: 10.1186/s13012-017-0590-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Ivers NM, Sales A, Colquhoun H, et al. No more “business as usual” with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci. 2014;9:14. doi: 10.1186/1748-5908-9-14. [DOI] [PMC free article] [PubMed] [Google Scholar]
15.Brehaut JC, Colquhoun HL, Eva KW, et al. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med. 2016;164:435–41. doi: 10.7326/M15-2248. [DOI] [PubMed] [Google Scholar]
16.Colquhoun HL, Squires JE, Kolehmainen N, Fraser C, Grimshaw JM. Methods for designing interventions to change healthcare professionals’ behaviour: a systematic review. Implement Sci 2017. 2017;12:30. doi: 10.1186/s13012-017-0560-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Grimshaw J, Ivers N, Linklater S, et al. on behalf of the Audit and Feedback MetaLab. Reinvigorating stagnant science: implementation laboratories and a meta-laboratory to efficiently advance the science of audit and feedback. BMJ Qual Saf. 2019 doi: 10.1136/bmjqs-2018-008355. [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Glazier RH, Klein-Geltink J, Kopp A, Sibley LM. Capitation and enhanced fee-for-service models for primary care reform: a population-based evaluation. CMAJ. 2009;180:E72–81. doi: 10.1503/cmaj.081316. [DOI] [PMC free article] [PubMed] [Google Scholar]
19.Jonah L, Pefoyo AK, Lee A, et al. Evaluation of the effect of an audit and feedback reporting tool on screening participation: the Primary Care Screening Activity Report (pcsar) Prev Med. 2017;96:135–43. doi: 10.1016/j.ypmed.2016.12.002. [DOI] [PubMed] [Google Scholar]
20.Michie S, Richardson M, Johnston M, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med. 2013;46:81–95. doi: 10.1007/s12160-013-9486-6. [DOI] [PubMed] [Google Scholar]
21.Collins LM, Kugler KC, Gwadz MV. Optimization of multicomponent behavioral and biobehavioral interventions for the prevention and treatment of hiv/aids. AIDS Behav. 2016;20(suppl 1):S197–214. doi: 10.1007/s10461-015-1145-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
22.Wyrick DL, Rulison KL, Fearnow-Kenney M, Milroy JJ, Collins LM. Moving beyond the treatment package approach to developing behavioral interventions: addressing questions that arose during an application of the Multiphase Optimization Strategy (most) Transl Behav Med. 2014;4:252–9. doi: 10.1007/s13142-013-0247-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
23.Bravo CA, Llovet D, Witteman HO, et al. Designing emails aimed at increasing family physicians’ use of a Web-based audit and feedback tool to improve cancer screening rates: cocreation process. JMIR Hum Factors. 2018;5:e25. doi: 10.2196/humanfactors.9875. [DOI] [PMC free article] [PubMed] [Google Scholar]
24.Vaisson G, Witteman HO, Bouck Z, et al. Testing behavior change techniques to encourage primary care physicians to access cancer screening audit and feedback reports: protocol for a factorial randomized experiment of email content. JMIR Res Protoc. 2018;7:e11. doi: 10.2196/resprot.9090. [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The precis-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147. doi: 10.1136/bmj.h2147. [DOI] [PubMed] [Google Scholar]
26.Canadian Institutes of Health Research (cihr), Natural Sciences and Engineering Research Council of Canada (nserc), and Social Sciences and Humanities Research Council of Canada (sshrc) TCPS2. Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans. Ottawa, ON: cihr, nserc, and sshrc; 2010. [Available online at: http://www.pre.ethics.gc.ca/pdf/eng/tcps2/tcps_2_final_web.pdf; cited 21 April 2019] [Google Scholar]
27.Canadian Institutes of Health Research (cihr), Natural Sciences and Engineering Research Council of Canada (nserc), and Social Sciences and Humanities Research Council of Canada (sshrc) The Consent Process. Ch. 3. Ottawa, ON: cihr, nserc, and sshrc; 2014. TCPS2 2014. Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans. [Available online at: http://www.pre.ethics.gc.ca/pdf/eng/tcps2-2014/TCPS_2_FINAL_Web.pdf; cited 3 December 2018] [Google Scholar]
28.Michie S, Atkins L, West R. The Behaviour Change Wheel : A Guide to Designing Interventions. Surrey, UK: Silverback Publishing; 2014. [Google Scholar]
29.International Standards Organization (iso) ISO 9241-210. Ergonomics of Human–System Interaction. Part 210: Human-Centred Design for Interactive Systems. Geneva, Switzerland: iso; 2010. [Available online at: https://www.sis.se/api/document/preview/912053; cited 27 February 2019] [Google Scholar]
30.Zou G. A modified Poisson regression approach to prospective studies with binary data. Am J Epidemiol. 2004;159:702–6. doi: 10.1093/aje/kwh090. [DOI] [PubMed] [Google Scholar]
31.Saunders B, Sim J, Kingstone T, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant. 2018;52:1893–907. doi: 10.1007/s11135-017-0574-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
32.Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13:117. doi: 10.1186/1471-2288-13-117. [DOI] [PMC free article] [PubMed] [Google Scholar]
33.Ivers NM, Grimshaw JM. Reducing research waste with implementation laboratories. Lancet. 2016;388:547–8. doi: 10.1016/S0140-6736(16)31256-9. [DOI] [PubMed] [Google Scholar]
34.Brown B, Gute WD, Blakeman T, et al. Clinical Performance Feedback Intervention Theory (cp-fit): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implement Sci. 2019 doi: 10.1186/s13012-019-0883-5. [In press] [DOI] [PMC free article] [PubMed] [Google Scholar]
35.Davidoff F, Dixon-Woods M, Leviton L, Michie S. Demystifying theory and its use in improvement. BMJ Qual Saf. 2015;24:228–38. doi: 10.1136/bmjqs-2014-003627. [DOI] [PMC free article] [PubMed] [Google Scholar]
36.Sheeran P, Webb TL. The intention–behavior gap. Soc Personal Psychol Compass. 2016;10:503–18. doi: 10.1111/spc3.12265. [DOI] [Google Scholar]
37.Montano DE, Phillips WR. Cancer screening by primary care physicians: a comparison of rates obtained from physician self-report, patient survey, and chart audit. Am J Public Health. 1995;85:795–800. doi: 10.2105/AJPH.85.6.795. [DOI] [PMC free article] [PubMed] [Google Scholar]
38.Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82:581–629. doi: 10.1111/j.0887-378X.2004.00325.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
39.Greenhalgh T, Wherton J, Papoutsi C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res. 2017;19:e367. doi: 10.2196/jmir.8775. [DOI] [PMC free article] [PubMed] [Google Scholar]
40.Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. doi: 10.1186/1748-5908-4-50. [DOI] [PMC free article] [PubMed] [Google Scholar]
41.Keith RE, Crosson JC, O’Malley AS, Cromp DA, Taylor EF. Using the Consolidated Framework for Implementation Research (cfir) to produce actionable findings: a rapid-cycle evaluation approach to improving implementation. Implement Sci. 2017;12:1–12. doi: 10.1186/s13012-017-0550-7. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

CONC-26-205-s001.pdf^{(1.1MB, pdf)}

[b1-conc-26-205] 1.Kerner J, Liu J, Wang K, et al. Canadian cancer screening disparities: a recent historical perspective. Curr Oncol. 2015;22:156–63. doi: 10.3747/co.22.2539. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b2-conc-26-205] 2.Anas R, Bell R, Brown A, Evans W, Sawka C. A ten-year history: the Cancer Quality Council of Ontario. Healthc Q. 2012;15(spec):24–7. doi: 10.12927/hcq.2012.23162. [DOI] [PubMed] [Google Scholar]

[b3-conc-26-205] 3.Cancer Care Ontario (cco) Ontario Cancer Screening Performance Report 2016. Toronto, ON: cco; 2016. [Google Scholar]

[b4-conc-26-205] 4.Wools A, Dapper EA, de Leeuw JRJ. Colorectal cancer screening participation: a systematic review. Eur J Public Health. 2016;26:158–68. doi: 10.1093/eurpub/ckv148. [DOI] [PubMed] [Google Scholar]

[b5-conc-26-205] 5.De Klerk CM, Gupta S, Dekker E, Essink-Bot ML on behalf of the Expert Working Group “Coalition to reduce inequities in colorectal cancer screening” of the World Endoscopy Organization. Socioeconomic and ethnic inequities within organised colorectal cancer screening programmes world-wide. Gut. 2018;67:679–87. doi: 10.1136/gutjnl-2016-313311. [DOI] [PubMed] [Google Scholar]

[b6-conc-26-205] 6.Peterson EB, Ostroff JS, DuHamel KN, et al. Impact of provider–patient communication on cancer screening adherence: a systematic review. Prev Med. 2016;93:96–105. doi: 10.1016/j.ypmed.2016.09.034. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b7-conc-26-205] 7.Anhang Price R, Zapka J, Edwards H, Taplin SH. Organizational factors and the cancer screening process. J Natl Cancer Inst Monogr. 2010;2010:38–57. doi: 10.1093/jncimonographs/lgq008. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b8-conc-26-205] 8.Cancer Care Ontario (cco) Who We Are [Web page] Toronto, ON: cco; 2015. [Available at: https://www.ccohealth.ca/en/who-we-are; cited 30 May 2017] [Google Scholar]

[b9-conc-26-205] 9.United States Department of Health and Human Services, Office of Disease Prevention and Health Promotion, Healthy People 2020. Cancer [Web page, Objectives tab] [Available at: https://www.healthypeople.gov/2020/topics-objectives/topic/cancer/objectives; cited 3 April 2019]

[b10-conc-26-205] 10.Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback : effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012:CD000259. doi: 10.1002/14651858.CD000259.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b11-conc-26-205] 11.Scott A, Sivey P, Ait Ouakrim D, et al. The effect of financial incentives on the quality of health care provided by primary care physicians. Cochrane Database Syst Rev. 2011:CD008451. doi: 10.1002/14651858.CD008451.pub2. [DOI] [PubMed] [Google Scholar]

[b12-conc-26-205] 12.Arditi C, Rège-Walther M, Wyatt JC, Durieux P, Burnand B. Computer-generated reminders delivered on paper to healthcare professionals; effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2012;12:CD001175. doi: 10.1002/14651858.CD001175.pub3. [DOI] [PubMed] [Google Scholar]

[b13-conc-26-205] 13.Tuti T, Nzinga J, Njoroge M, et al. A systematic review of electronic audit and feedback: intervention effectiveness and use of behaviour change theory. Implement Sci. 2017;12:61. doi: 10.1186/s13012-017-0590-z. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b14-conc-26-205] 14.Ivers NM, Sales A, Colquhoun H, et al. No more “business as usual” with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci. 2014;9:14. doi: 10.1186/1748-5908-9-14. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b15-conc-26-205] 15.Brehaut JC, Colquhoun HL, Eva KW, et al. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med. 2016;164:435–41. doi: 10.7326/M15-2248. [DOI] [PubMed] [Google Scholar]

[b16-conc-26-205] 16.Colquhoun HL, Squires JE, Kolehmainen N, Fraser C, Grimshaw JM. Methods for designing interventions to change healthcare professionals’ behaviour: a systematic review. Implement Sci 2017. 2017;12:30. doi: 10.1186/s13012-017-0560-5. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b17-conc-26-205] 17.Grimshaw J, Ivers N, Linklater S, et al. on behalf of the Audit and Feedback MetaLab. Reinvigorating stagnant science: implementation laboratories and a meta-laboratory to efficiently advance the science of audit and feedback. BMJ Qual Saf. 2019 doi: 10.1136/bmjqs-2018-008355. [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]

[b18-conc-26-205] 18.Glazier RH, Klein-Geltink J, Kopp A, Sibley LM. Capitation and enhanced fee-for-service models for primary care reform: a population-based evaluation. CMAJ. 2009;180:E72–81. doi: 10.1503/cmaj.081316. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b19-conc-26-205] 19.Jonah L, Pefoyo AK, Lee A, et al. Evaluation of the effect of an audit and feedback reporting tool on screening participation: the Primary Care Screening Activity Report (pcsar) Prev Med. 2017;96:135–43. doi: 10.1016/j.ypmed.2016.12.002. [DOI] [PubMed] [Google Scholar]

[b20-conc-26-205] 20.Michie S, Richardson M, Johnston M, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med. 2013;46:81–95. doi: 10.1007/s12160-013-9486-6. [DOI] [PubMed] [Google Scholar]

[b21-conc-26-205] 21.Collins LM, Kugler KC, Gwadz MV. Optimization of multicomponent behavioral and biobehavioral interventions for the prevention and treatment of hiv/aids. AIDS Behav. 2016;20(suppl 1):S197–214. doi: 10.1007/s10461-015-1145-4. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b22-conc-26-205] 22.Wyrick DL, Rulison KL, Fearnow-Kenney M, Milroy JJ, Collins LM. Moving beyond the treatment package approach to developing behavioral interventions: addressing questions that arose during an application of the Multiphase Optimization Strategy (most) Transl Behav Med. 2014;4:252–9. doi: 10.1007/s13142-013-0247-7. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b23-conc-26-205] 23.Bravo CA, Llovet D, Witteman HO, et al. Designing emails aimed at increasing family physicians’ use of a Web-based audit and feedback tool to improve cancer screening rates: cocreation process. JMIR Hum Factors. 2018;5:e25. doi: 10.2196/humanfactors.9875. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b24-conc-26-205] 24.Vaisson G, Witteman HO, Bouck Z, et al. Testing behavior change techniques to encourage primary care physicians to access cancer screening audit and feedback reports: protocol for a factorial randomized experiment of email content. JMIR Res Protoc. 2018;7:e11. doi: 10.2196/resprot.9090. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b25-conc-26-205] 25.Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The precis-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147. doi: 10.1136/bmj.h2147. [DOI] [PubMed] [Google Scholar]

[b26-conc-26-205] 26.Canadian Institutes of Health Research (cihr), Natural Sciences and Engineering Research Council of Canada (nserc), and Social Sciences and Humanities Research Council of Canada (sshrc) TCPS2. Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans. Ottawa, ON: cihr, nserc, and sshrc; 2010. [Available online at: http://www.pre.ethics.gc.ca/pdf/eng/tcps2/tcps_2_final_web.pdf; cited 21 April 2019] [Google Scholar]

[b27-conc-26-205] 27.Canadian Institutes of Health Research (cihr), Natural Sciences and Engineering Research Council of Canada (nserc), and Social Sciences and Humanities Research Council of Canada (sshrc) The Consent Process. Ch. 3. Ottawa, ON: cihr, nserc, and sshrc; 2014. TCPS2 2014. Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans. [Available online at: http://www.pre.ethics.gc.ca/pdf/eng/tcps2-2014/TCPS_2_FINAL_Web.pdf; cited 3 December 2018] [Google Scholar]

[b28-conc-26-205] 28.Michie S, Atkins L, West R. The Behaviour Change Wheel : A Guide to Designing Interventions. Surrey, UK: Silverback Publishing; 2014. [Google Scholar]

[b29-conc-26-205] 29.International Standards Organization (iso) ISO 9241-210. Ergonomics of Human–System Interaction. Part 210: Human-Centred Design for Interactive Systems. Geneva, Switzerland: iso; 2010. [Available online at: https://www.sis.se/api/document/preview/912053; cited 27 February 2019] [Google Scholar]

[b30-conc-26-205] 30.Zou G. A modified Poisson regression approach to prospective studies with binary data. Am J Epidemiol. 2004;159:702–6. doi: 10.1093/aje/kwh090. [DOI] [PubMed] [Google Scholar]

[b31-conc-26-205] 31.Saunders B, Sim J, Kingstone T, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant. 2018;52:1893–907. doi: 10.1007/s11135-017-0574-8. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b32-conc-26-205] 32.Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13:117. doi: 10.1186/1471-2288-13-117. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b33-conc-26-205] 33.Ivers NM, Grimshaw JM. Reducing research waste with implementation laboratories. Lancet. 2016;388:547–8. doi: 10.1016/S0140-6736(16)31256-9. [DOI] [PubMed] [Google Scholar]

[b34-conc-26-205] 34.Brown B, Gute WD, Blakeman T, et al. Clinical Performance Feedback Intervention Theory (cp-fit): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implement Sci. 2019 doi: 10.1186/s13012-019-0883-5. [In press] [DOI] [PMC free article] [PubMed] [Google Scholar]

[b35-conc-26-205] 35.Davidoff F, Dixon-Woods M, Leviton L, Michie S. Demystifying theory and its use in improvement. BMJ Qual Saf. 2015;24:228–38. doi: 10.1136/bmjqs-2014-003627. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b36-conc-26-205] 36.Sheeran P, Webb TL. The intention–behavior gap. Soc Personal Psychol Compass. 2016;10:503–18. doi: 10.1111/spc3.12265. [DOI] [Google Scholar]

[b37-conc-26-205] 37.Montano DE, Phillips WR. Cancer screening by primary care physicians: a comparison of rates obtained from physician self-report, patient survey, and chart audit. Am J Public Health. 1995;85:795–800. doi: 10.2105/AJPH.85.6.795. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b38-conc-26-205] 38.Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82:581–629. doi: 10.1111/j.0887-378X.2004.00325.x. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b39-conc-26-205] 39.Greenhalgh T, Wherton J, Papoutsi C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res. 2017;19:e367. doi: 10.2196/jmir.8775. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b40-conc-26-205] 40.Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. doi: 10.1186/1748-5908-4-50. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b41-conc-26-205] 41.Keith RE, Crosson JC, O’Malley AS, Cromp DA, Taylor EF. Using the Consolidated Framework for Implementation Research (cfir) to produce actionable findings: a rapid-cycle evaluation approach to improving implementation. Implement Sci. 2017;12:1–12. doi: 10.1186/s13012-017-0550-7. [DOI] [PMC free article] [PubMed] [Google Scholar]

PERMALINK

Testing e-mail content to encourage physicians to access an audit and feedback tool: a factorial randomized experiment

G Vaisson, BSc MSc

HO Witteman, PhD

S Chipenda-Dansokho, PhD

M Saragosa, RN MN

Z Bouck, MPH

CA Bravo, MSc

L Desveaux, PhD

D Llovet, MA PhD

J Presseau, BA MRes PhD

M Taljaard, PhD

S Umar, BA MSc

JM Grimshaw, MBChB PhD

J Tinmouth, MD PhD

NM Ivers, MD PhD

Abstract

Background

Methods

Results

Conclusions

INTRODUCTION

METHODS

Experimental Design, Setting, and Eligibility Criteria

Interventions

Baseline Characteristics

Outcomes

Data Collection

Power

Randomization, Allocation Concealment, and Blinding

Statistical Analysis

Process Evaluation

RESULTS

Participant Flow

FIGURE 1.

Baseline Characteristics

TABLE I.

Screening Activity Report Access by Factor

TABLE II.

Main Effects on Screening Activity Report Access

TABLE III.

Main Effects on the Number of Times the Screening Activity Report Was Accessed

E-Mail Message Open and Click Rates

Main Effects on Up-to-Date Breast, Cervical, and Colorectal Cancer Screening Status

Ancillary Analyses

Exploratory Interactions and Modification Effects

TABLE IV.

Process Outcomes by Factor

Process Evaluation

TABLE V.

DISCUSSION

Reflections on the Value of Catalyst Project Funding

CONCLUSIONS

Supplemental Materials

ACKNOWLEDGMENTS

Footnotes

REFERENCES

Associated Data

Supplementary Materials

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases