Effect of hand versus electronic signatures on response rates in postal surveys: a randomised controlled trial among emergency physicians in Canada

Abstract

Objectives

Hand signatures offer a more authentic personalisation, which carries over to a sense of trust, although are costly and time-consuming when considering large postal surveys. The objective of this study was to compare response rates when using either hand-signed or electronic-signed letters in a postal survey.

Design and setting

We embedded this randomised controlled trial within a national cross-sectional postal survey of emergency physicians in Canada. The survey aimed to describe current practice patterns with respect to primary headache disorders.

Participants

We randomly sampled 500 emergency physicians listed in the Scott’s Canadian Medical Directory, 2019 edition.

Interventions

Using computer-generated random numbers, physicians were allocated to receiving either hand-signed (n=250) or electronic signed (n=250) letters. The initial mailout contained a US$5 Tim Hortons coffee card with the invitation letter. Four reminders were sent to non-responders every 3 weeks. The same type of signature was used for the initial invitation and subsequent reminders.

Outcome

The primary outcome was the survey response rate.

Results

Among 500 physicians invited, 32 invitations were undeliverable. Among the remaining 468 physicians, 231 had been allocated to the hand-signed group and 237 to the electronic signed group. The response rate in the hand-signed group was 87 (37.7%) vs 97 (40.9%) in the electronic-signed group (absolute difference in proportions −3.3%, 95% CI −12.1% to 5.6%).

Conclusion

There was no significant difference in physician response rate between hand-signed and e-signed cover letter and reminder letters. Electronic signatures should be used in future postal surveys among physicians to save on time and labour without impacting response rates.

Strengths and limitations of this study.

• Our survey methodology used a modified Dillman’s tailored design with additional steps to enhance response rates, such as incentives and removal of prenotification letters.

• The survey questionnaire underwent rigorous testing prior to distribution using cognitive interviews on emergency physicians.

• The survey was pilot tested to local addresses to ensure no issues with our postal procedure.

• We obtained a relatively modest response rate, which may be attributed to the COVID-19 pandemic.

Introduction

Background

Physician surveys are often used to obtain information about their perspectives and attitudes towards clinical problems. A major challenge with physician surveys is obtaining an adequate response rate. Postal surveys typically have a higher response rate (up to 20% higher) compared with other modes of administration such as internet-based surveys, but are more costly and labour intensive.^{1 2} There is evidence to suggest that physician response rates have been declining with time.^{1 3} Among the reasons for declining response rates are lack of time during core working hours and gatekeepers such as receptionists who may perceive the survey as irrelevant and prevent it from reaching the recipient.¹ Exploring avenues to optimise response rates with respect to labour and cost is important.

The Dillman’s Tailored Design Method provides recommendations on the optimal construction of surveys using various modes of administration.⁴ These methods are well established in the literature and have shown response rates ranging from 50% to 65% when used as stand-alone mode of survey administration.⁴ Methods for optimising postal surveys are continuously being explored to lower expenses and reduce labour. In our experience, the costs and labour involved in a postal survey include (1) printing large amounts of paper for the survey instrument and supporting documentation; (2) costs associated with printing and purchasing envelopes; (3) costs associated with prestamping the prepaid envelopes for each contact and sending the overall package; (4) folding printed materials into thirds and inserting into envelopes; (5) manual data entry and (6) hand signing of recruitment and reminder letters.

The effects of personalisation on response rates has been previously explored: insertion of name and address and blue ink signatures in each letter compared with mass-copied letters with group salutations improved response rates from 3% to 12% for the general population.⁵ It is unclear if these findings hold true in a specialised population of physicians. Although it is more labour intensive to hand-sign masses of letters for each contact point, hand signatures have a more attractive visual appeal, offer a more personalised effect and carry over to a sense of trust which may contribute to a higher response rate.⁶ On the other hand, if hand-signed letters show no significantly higher response rate than electronically signed (e-signed) letters, this would be useful information and could reduce time and labour in future postal survey administration.

Objectives

The primary objective of this randomised trial embedded within a large national postal survey was to determine if hand-signed letters resulted in a higher response rate compared with e-signed letters among Canadian emergency physicians.

We previously found some differences in response rates between Canadian French-speaking and English-speaking participants.⁷ We; therefore, conducted a post hoc exploratory subgroup analyses to compare the effect of e-signatures versus hand signatures between English-speaking and Canadian French-speaking participants.

Methods

Study design and participants

We used a national self-administered postal survey of emergency physicians listed in the 2019 Scott’s Canadian Medical Directory, which is Canada’s leading source for contact information and claims to list over 98% of practising physicians with 97% address accuracy.⁸ The results from the survey are reported elsewhere.⁷ The survey (online supplemental appendix A) was mailed to a random sample of 500 physicians. Physicians were eligible for the survey if they were currently treating adults in emergency medicine. The questionnaire was two pages in length and consisted of 12 questions which took approximately 10 min to complete. The survey addressed emergency physicians’ current practice for treating benign headache disorders in the emergency department and their perspectives and attitudes towards peripheral nerve blocks (PNBs). PNBs are minor bedside procedures which are sometimes used to treat primary headache disorders in the emergency department. The survey questionnaire underwent rigorous development, using cognitive interviews, on 10 practising emergency physicians in English and Canadian French. The interviewer directly observed physicians complete the survey as they read questions aloud and identified any verbal and non-verbal cues which signified confusion or hesitation. The survey was adjusted after each iteration with respect to the content, organisation, grammar and overall layout to arrive at a finalised survey questionnaire.

Patient and public involvement

Neither patients nor the general public were involved in any formal way with this study.

Intervention

Using computer-generated numbers in Microsoft Excel (Redmond, Washington, USA), the principal investigator randomly assigned half (n=250) of our random sample of 500 emergency physicians to receive hand-signed letters and the remaining half to receive e-signed letters. The letters at each mailout (initial mailout and reminders 1–4) were all either hand-signed or e-signed depending on each emergency physicians assigned group. In the hand-signature group, the same two investigators (DP and JJP) signed each letter in blue ink above printed names, credentials and affiliations. In the e-signed group, scanned electronic signatures (in black ink) of DP and JJP were placed above printed name, credentials and affiliations at the bottom of each letter (online supplemental appendix B).

Outcome measure

The primary outcome was the response rate, where a response was considered any physician who returned a partially or fully completed questionnaire. The denominator was considered all successfully delivered surveys (ie, the number invited minus those returned undeliverable).

Survey administration

The survey was administered according to a modified Dillman’s method, including an initial mailout with an unconditional US$5 Tim Horton’s coffee card, either a hand-signed or e-signed recruitment letter, the survey instrument, an information sheet and a postage-paid envelope. We used four reminders sent every 3 weeks to non-responders with either hand-signed or e-signed reminder letters stating the number of weeks it has been since the last letter with a new survey instrument and a prepaid return envelope. The final reminder was sent using Canada Post Xpresspost, which guarantees delivery within 1–3 business days and is larger and more visually appealing in appearance. Surveys were administered in English or Canadian French according to recipients’ language preference as reported by the Canadian Medical Directory. Prior to sending the initial mailout, we pilot tested (n=20) our survey to local addresses in June 2021 to ensure no issues with our postal procedure. The initial mailout (n=480) was sent in July 2021 and the last contact was made in October 2021. We used additional measures in attempt to resend undeliverable letters by searching the physician in the College of Physicians and Surgeons of Ontario (or equivalent for the respective jurisdiction) for an updated primary practice location (online supplemental appendix C).

Sample size rationale

This trial was embedded within an existing survey and the sample size was therefore predetermined by the objectives of the survey. We, therefore, calculated the detectable difference given the available sample size, which was sufficient to detect an absolute difference between the hand-signed and e-signed groups response rates of 13% with 80% power, assuming a response rate of 50% in the e-signed group and using a two-sided test at the 5% level of significance. Previous surveys using similar methods have achieved response rates around 50%.^9–11

Data analysis

Data from returned survey questionnaires were manually entered into Microsoft Excel (Redmond, Washington, USA). We assessed differences between respondents and non-respondents in geographical practice location in Canada and language preference using χ² tests. We used a χ² test to compare the response rates in the hand-signature and e-signed groups together with absolute and relative difference in proportions and a two-sided 95% CI. All data analyses were conducted using Microsoft Excel (Redmond, Washington, USAs) and SAS V.9.2 (SAS Institute).

For the exploratory subgroup analysis, we stratified results by language and obtained absolute differences in proportions. We also tested for a significant interaction effect using logistic regression.

Results

Response rate

Figure 1 demonstrates the flow diagram for our study. We launched the survey in June 2021 and collected final responses in January 2022. From the 2955 emergency physicians listed in threvision of the funding policy, having modelse Canadian Medical Directory, we randomly selected 500 and assigned n=250 into each of the hand-signed and e-signed groups. Thirty-two surveys were returned undeliverable due to change in practice location, retired or other reasons. Of 468 delivered surveys (231 in the hand-signed and 237 in the e-signed group), we received 184 responses for an overall response rate of 39.3%. The response rate in the hand-signed group was 87 (37.7%) compared with 97 (40.9%) in the e-signed group (absolute difference in proportions −3.3%. 95% CI −12.1% to 5.6%, relative difference: 0.92, 95% CI 0.73 to 1.15).

Figure 1.

Participants and response rate in hand-signature group versus electronic-signature group.

Respondent characteristics

Table 1 displays respondent demographics in the hand-signed and e-signed groups. Physician demographics were similar between the hand-signature and e-signed group. The majority of the respondents were male (65.5%), in practice for 10 or more years (78.4%) and had a Canadian College of Family Physicians with specialisation in Emergency Medicine designation (51.9%). Most emergency physicians practised in Ontario (40.8%), Western Canada (31.5%) and Quebec (19.6%) and in an academic health centre or community/district teaching hospital (83.1%), which accommodates 60 000 or more emergency department visits per year (56.8%).

Table 1.

Demographics and practice setting among hand-signature and electronic-signature groups and eligible emergency physicians

Characteristic¶	Hand-signature N (%)	Electronic-signature N (%)
Gender	N=70	N=78
Female	27 (38.6)	23 (29.5)
Male	43 (61.4)	54 (69.2)
Other	0 (0)	0 (0)
Prefer not to say	0 (0)	1 (1.3)
Language*	N=87	N=97
English	72 (82.8)	80 (82.4)
Canadian French	15 (17.2)	17 (17.5)
Years of practice	n=70	n=78
1–4	3 (4.3)	2 (2.6)
5–9	12 (17.1)	12 (15.4)
10–19	26 (37.1)	31 (39.7)
≥20	26 (41.4)	33 (42.3)
Region†	n=87	n=97
Western Canada	23 (26.4)	35 (36.1)
Ontario	35 (40.2)	40 (41.2)
Quebec	19 (21.8)	17 (17.5)
Eastern Canada	10 (11.5)	5 (5.2)
Canadian professional designation‡	n=60	n=69
CCFP	1 (1.7)	1 (1.4)
CCFP-EM	32 (53.3)	35 (50.7)
FRCPC-EM	14 (23.3)	18 (26.1)
Multiple§	13 (21.7)	12 (17.4)
Other	0 (0)	3 (4.3)
Practice setting	n=62	n=68
Academic health centre	21 (33.8.)	20 (29.4)
Community/district general hospital: teaching	30 (48.4)	37 (54.4)
Community/district general hospital: non-teaching	9 (14.5)	8 (11.7)
Rural	2 (3.2)	3 (4.4)
Other	0 (0)	1 (1.5)
Patient visits to ED per year	n=69	n=77
<30 000	12 (17.4)	7 (9.1)
30 000–59 999	19 (27.5)	25 (32.5)
60 000–79 999	22 (31.9)	24 (31.2)
>80 000	16 (23.2)	21 (27.3)

Eastern Canada: Prince Edward Island, Nova Scotia, Newfoundland & Labrador, New Brunswick.

*Region and Language preference: for all survey respondents; all other demographics are for eligible participants. Eligibility criteria were those currently practicing adult emergency medicine.

†Region: Western Canada: British Columbia, Alberta, Saskatchewan and Manitoba.

‡CCFP: Canadian College of Family Physicians; CCFP-EM: CCFP with a specialisation in Emergency Medicine; FRCPC-EM: Fellow of the Royal College of Physicians and Surgeons of Canada in Emergency Medicine.

§Multiple: physician holds more than one of the above designations.

¶The variation in denominator for each variable is due to missing or unanswered responses.

ED, Emergency Department.

For the exploratory subgroup analysis comparing effect of the type of signatures between English-speaking and Canadian French-speaking participants (table 2), the response rate among English-language participants was 72/201 (35.8%) in the hand-signed group compared with 80/200 (40%) in the e-signed group (absolute difference: −4.2%, 95% CI −13.7% to 5.6%). Among Canadian French-speaking participants, the response rate was 15/31 (48.4%) in the hand-signed group compared with 17/36 (47.2%) in the e-signed group (absolute difference: 1.2%, 95% CI −22.8% to 25.2%); p value for statistical interaction p=0.68.

Table 2.

Response rates in hand-signature vs electronic-signature groups, stratified by language preference

Language preference	Hand-signature	Electronic-signature	Absolute difference (95% CI)	P value*
				0.68
English	35.8%	40%	−4.2% (−13.7% to 5.6%)
Canadian French	48.4%	47.2%	1.2% (−22.8% to 25.2%)
Overall	37.7%	40.9%	−3.3% (−12.1% to 5.6%)

*P value derived from logistic regression interaction term between signature type and language preference.

Discussion

Our randomised controlled trial embedded within a national postal survey was unable to demonstrate that hand-signatures have a higher response rate than e-signatures. This is an important finding for the methodology of future postal surveys as the time involved with hand-signing can be replaced with other less time-consuming methods such as e-signatures without negatively affecting response rates. In our survey, both investigators signed approximately 825 letters each, which involved several hours of monotonous labour. Based on our response rate and with 4 reminders to non-respondents, approximately 1825 signatures would be required if all respondents were to receive hand-signatures. Although hand-signatures appear valuable, this step is time-consuming, and our results suggest this step may be replaced with e-signatures thus saving several hours of time and labour. Although we found no statistically significant difference, we note that the response rate was lower in the hand-signature group compared with the e-signature group, which was unexpected. Hand-signatures offer a more authentic method of personalisation which may carry over to sense of trust, compared with e-signatures which may be viewed as less trusted and decrease acceptance;⁶ we expected a higher response rate among this group. The electronic signatures were a bold, black colour, whereas the hand-signatures were signed using blue ink. The higher response rate in the electronic signature group may have been due to chance; further research is needed with a larger sample size to better understand these results.

A meta-analysis of randomised controlled trials of strategies that influence response rates to postal surveys among various disciplines, found a higher response rate among more personalised appearing letters compared with less personalised letters (OR 1.16, 95% CI 1.06 to 1.28);¹² however, only 5 of 48 studies included in the analysis compared hand-signatures against printed signatures.^12–14 A previous national postal survey conducted in 1999 with overall response rate of 78.7% found no significant difference between the hand signed group and computer printed group (relative difference 1.01, 95% CI 0.98 to 1.04).¹³ Their method of postal administration differed from ours in that they only used two reminders, no incentives and no Xpress post (courier like delivery) in their final reminder. They did, however, use personal salutations whereas ours did not. Their study was conducted in the UK to members of the Royal College of Obstetricians. Our study is 20 years more recent and applicable to future postal surveys, especially for a specialised population of emergency physicians working in busy emergency departments across Canada. Additionally, an electronic signature in present-time looks similar to physical hand-signatures, compared with electronic signatures from the 20th century.

Our study has several strengths. Our methods were adapted and improved from previous postal surveys using the Dillman’s technique, such as inclusion of an unconditional US$5 Tim Horton’s coffee card with the initial mailout which has been shown to significantly improve response rates¹⁵ and removal of prenotification letters which have shown to result in lower response rates in a study using similar methods to ours.¹⁶ We also used additional measures to ensure physicians who have moved primary practice locations received their survey by verifying with the appropriate provincial regulatory body’s website (eg, College of Physicians and Surgeons of Ontario) of any updates to primary practice location before attempting to resend letters that were undeliverable and returned to sender. Previous surveys have not documented this additional step and we found this beneficial in improving our response rate. Our findings may be generalisable to emergency physicians across Canada since the Canadian Medical Directory lists 99% of all practising physicians in Canada. No other database contains postal addresses for Canadian emergency physicians.

Our study has some limitations. Despite using these rigorous methods, we obtained a relatively modest response rate of 39.3%; however, this is higher than other recent surveys among emergency physicians.^{17 18} The response rate may be attributed to several factors such as influence from the COVID-19 pandemic, which is known to contribute increased burden and stress on Canadian emergency departments, resulting in less time and perhaps less interest for emergency physicians to complete postal surveys. Research has shown reduced response rates to certain medical surveys during the COVID-19 pandemic as a result of survey fatigue.¹⁹ Additionally, there was an increased hesitancy among the general population with touching foreign materials, including mail, in fear of virus transmission. The survey was conducted from June 2021 to December 2021, in the midst of the pandemic. Furthermore, the Canadian Medical Directory is a common source for other postal surveys; thus, there is a possibility of overlap with other studies. Physicians who receive multiple consecutive surveys may find this too overwhelming and may be less likely to respond. Finally, the detectable difference in our study was relatively large; we were inadequately powered to detect smaller but meaningful differences.

Our study contributes to an important and labour-intensive aspect of the methodology of postal surveys. Future studies should continue to study the effects of personalisation using our methods and include hand-written or e-signed salutations compared with hand-written or e-signed generic salutations to determine if this would enhance response rates.

Conclusion

There was no significant difference in physician response rate between hand-signed and e-signed cover letter and reminder letters. Electronic signatures should be used in future postal surveys among physicians to save on time and labour without impacting response rates.

Data availability statement

Data are available on reasonable request.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

This study was approved by the Ottawa Health Science Network Research Ethics Board (protocol ID: 20210212-01H). All participants of the survey received an information sheet stating that participation in the study was voluntary and completion of the questionnaire indicated consent to participate in the study.