The Effect of Informed Consent on Results of a Standard Upper Extremity Intake Questionnaire

Abstract

Purpose

The trend toward requiring explicit consent from patients participating in observational research introduces the potential for consent bias, either through selection bias or through the Hawthorne effect. In the Hawthorne effect, patients may alter responses based on awareness of participation in a study, thus potentially limiting its applicability to a generalized orthopedic practice. We hypothesized that study subjects’ awareness of participation in an observational study by informed consent would alter responses to a standard upper extremity questionnaire in a statistically and clinically meaningful way compared with patients who filled out the questionnaire as a quality control measure.

Methods

We retrospectively reviewed data on 39 patients who underwent isolated carpal tunnel release, who had completed preoperative and postoperative Quick Disabilities of the Arm, Shoulder, and Hand (Quick DASH) forms without providing consent for study participation. Next, we approached 35 patients scheduled to undergo isolated carpal tunnel release who completed the intake questionnaire on the day of surgery, for consent to participate in the study. After obtaining consent but before surgery, these patients completed a second questionnaire and then completed a postoperative questionnaire at a mean of 8 weeks postoperatively.

Results

There were no significant differences in age, sex, insurance status, symptom duration, nerve conduction study and electromyography results, or disease severity between groups. We identified no statistically significant difference in preoperative or postoperative Quick DASH score between the retrospective and prospective cohorts (40 + 23 vs 40 + 19 preoperatively; 27 + 25 vs 19 + 13 postoperatively) or within the prospective cohort before and after obtaining informed consent.

Conclusions

Informed consent did not significantly alter patients’ responses to the Quick DASH questionnaire. These results suggest that both opt-in and opt-out approaches to observational research in hand surgery provide results that may be applicable to a generalized orthopedic practice.

Clinical relevance

This study provides evidence that will inform the interpretation of observational research findings in hand surgery.

Requiring explicit consent from patients participating in observational research increases the time and resources required to perform such research. The requirement for explicit consent may introduce the potential for selection bias, or consent bias, traditionally defined as differences in patient characteristics between consenters and nonconsenters.¹ Several authors have demonstrated that the informed consent process introduces a systemic bias in which nonconsenters and consenters have significant differences in estimation of disease prevalence, disease severity, clinical parameters, insurance status, and socioeconomic status.^2–5 These authors argue that using an opt-in approach, in which a patient must actively consent to participate in a study, compared with an opt-out approach, in which patients’ data may be included in a study (ie, database or registry) without explicit consent, limits the applicability of a research result to a generalized population.

The informed consent process may also introduce another form of bias: one in which study participants’ behavior changes based on awareness of participation in a study. This is known as the Hawthorne effect.⁶ This study attempted to define the effect of informed consent on patients’ responses to a standard questionnaire in a hand surgery practice.

A paucity of medical literature addresses the Hawthorne effect, and controversy exists as to whether the effect actually exists.⁷ McCarney et al⁸ evaluated a cohort of patients with mild to moderate dementia enrolled in a placebo-controlled trial of Ginkgo biloba. The investigators found a significantly improved measure of cognitive function at 6 months in patients who received intensive follow-up (baseline and 2-mo, 4-mo, and 6-mo follow-up) compared with patients who received minimal follow-up (baseline and 6-mo follow-up only). Conversely, Fernald et al⁹ found no evidence of a Hawthorne effect in a study of patients with skin and soft tissue infections.

We hypothesized that study subjects’ awareness of participation in an observational study by informed consent would alter questionnaire responses compared with patients completing the questionnaire as a quality control measure. A difference in outcome would support the argument for use of an opt-out approach, whereas a negative result would support the validity of either opt-in or opt-out approaches to observational research in hand surgery.

MATERIALS AND METHODS

The University of Rochester institutional review board reviewed and approved the research design.

Patient selection

Patients 18 years of age and older with a diagnosis of carpal tunnel syndrome who underwent primary carpal tunnel release were eligible for inclusion in the study. For inclusion, patients must have undergone an electromyography/nerve conduction study with findings consistent with isolated carpal tunnel syndrome. We excluded patients undergoing revision surgery, as well as patients with any other comorbid condition affecting either upper extremity. Data collected via chart review included age, sex, occupation, insurance status, symptom duration, and electrodiagnostic findings. We excluded patients with electrodiagnostic findings other than carpal tunnel syndrome.

For the retrospective cohort, we identified 39 patients who underwent isolated carpal tunnel release between January and August 2011. The patients included in the study all completed an intake and postoperative Quick Disabilities of the Arm, Shoulder, and Hand (DASH) questionnaire. A total of 25 patients completed preoperative questionnaires on the day of surgery and 14 completed the questionnaire at the initial visit (mean 103 d preoperatively; range, 10–366 d).

For the prospective cohort, 35 patients enrolled in the study on the day of surgery. We obtained informed consent using a consent form approved by the institutional review board, which specified that the purpose of the study was to ascertain differences in questionnaire responses when subjects were made aware that they were participating in a study. No patients refused consent for this study. All patients had previously completed the intake form. A total of 27 patients completed preoperative questionnaires on the day of surgery and 8 completed the questionnaire at the initial visit (mean 97 d preoperatively; range, 15–372 d). Immediately after the consent process, patients completed a second preoperative Quick DASH form. Patients who completed the intake questionnaire on the day of surgery then completed the second preoperative questionnaire approximately 1 hour after completing the initial questionnaire. These patients subsequently completed a postoperative questionnaire either at the time of clinic follow-up or via telephone if the patient did not follow up at an appropriate time. All follow-up data were completed at an average of 8 to 10 weeks postoperatively. Three patients were lost to follow-up.

Nerve conduction study/electromyography

Nerve conduction study results were available for all patients. All patients had isolated carpal tunnel syndrome. Six reports from outside institutions did not include the motor and sensory latency data in the retrospective cohort, and 1 patient in the prospective cohort did not have a report available because the patient worked as an electromyography technician and was tested without a formal report. Exclusion of this patient’s data did not alter our results; therefore, we included the patient’s data in the final analysis.

We graded carpal tunnel syndrome severity based on electrodiagnostic findings as previously described.¹⁰ Patients with slowing of conduction (either sensory or motor) were grade 1, whereas slowing of both fiber types was grade 2. Findings of thenar muscle denervation in the presence of slowed conduction represented grade 3. Evidence of diffuse but incomplete denervation was grade 4, and complete absence of response was grade 5.¹⁰

Outcomes questionnaire

The DASH is a validated outcome measure used to compare presurgical and postsurgical level of function in patients with conditions affecting the upper extremity, including carpal tunnel syndrome.^11–16 The minimal clinically important difference for the DASH has been previously established to be 10 points on a scale of 0 to 100, where 100 is the worst possible score.¹⁵ The Quick DASH questionnaire is a shortened version (11 questions) of the DASH questionnaire (30 questions) used to assess severity of symptoms and difficulty carrying out specific tasks with the upper extremity.¹⁶ The reliability, validity, and responsiveness of Quick DASH are comparable to those of the original DASH questionnaire and are commonly used because of relative ease of completion by patients.^17,18

We previously instituted the Quick DASH form as a standardized new patient intake questionnaire for all new and postoperative patients in the upper extremity practices of 2 surgeons before the initiation of this study. These patients did not explicitly consent to study participation. Because of variation in protocol between surgeons, most patients from 1 practice received the Quick DASH at the time of initial visit, and patients in another practice received the Quick DASH in the waiting room on the day of surgery.

We made 2 primary comparisons. First, we compared the retrospective cohort intake (preoperative) and postoperative questionnaire responses with the postconsenting preoperative and postoperative questionnaire responses in the prospectively enrolled group. Second, we compared the preconsent questionnaire responses with the questionnaire responses obtained immediately after obtaining informed consent, to serve as an internal control study arm.

Statistical analysis

The primary outcome measure was difference in Quick DASH score. Based on the established minimal clinically important difference, power analysis determined that 32 patients per group were necessary to detect a 10-point difference in Quick DASH score given an alpha error of 0.05 and beta error of 0.8.

We carried out statistical analysis using standard functions in Microsoft Excel (Redmond, WA). We analyzed categorical data using Fisher’s exact test. For continuous variables, we checked samples for normalcy of distribution. For this study, samples were not normally distributed; therefore, we used the Mann-Whitney test for statistical analysis of continuous variables. A biostatistician reviewed the experimental design, results, and analysis.

RESULTS

We included 39 patients in the retrospective cohort, and 35 patients were prospectively enrolled. Table 1 lists the demographic characteristics. Although there was a difference in the number of Medicare and Medicaid patients in each group, there were no statistically significant differences in these parameters. There were also no statistically significant differences in the electromyography/nerve conduction study findings or carpal tunnel severity score between the 2 cohorts.

TABLE 1.

Demographics and Carpal Tunnel Characterization

	Retrospective Cohort (n = 39)	Prospective Cohort (n = 35)	P Value*
Age (SD) (y)	56 (13)	56 (15)	.76
Sex
Male	13 (33%)	14 (40%)	.63
Insurance
Privately insured	25	21	.82
Medicare	5	10	.15
Medicaid	5	1	.09
Workers’ compensation	4	3	.70
Duration of symptoms (mo)	31 (37)	32 (35)	.49
Nerve conduction test results (SD) (n)	39	34
Motor latency above normal (ms)	1.5 (2.0)	1.5 (1.9)	.82
No motor response detected	1	0	> .99
Sensory latency above normal (ms)	0.9 (1.3)	0.5 (1.4)	.15
No sensory response detected	9	10	.61
Electromyelogram abnormal	11	11	.80
Carpal tunnel severity (1–5)	2.5 (0.7)	2.5 (0.9)	.92

^*Determined by Fisher’s exact test for categorical variables and the Mann-Whitney test for continuous variables.

We compared preoperative and postoperative Quick DASH scores of the retrospective (no consent) and prospective (consented) cohorts (Table 2). Postoperatively, patients in the retrospective cohort completed the questionnaire at 10 + 4 weeks (range, 4–17 wk), and the prospective cohort completed the questionnaire at 8 + 2 weeks (range, 4–14 wk) (P = .06). Preoperatively, the retrospective and prospective groups had similar Quick DASH scores. Postoperatively, mean Quick DASH differences were not significantly different between groups.

TABLE 2.

Retrospective Versus Prospective (Consented) Cohorts’ Quick DASH Response (Mean ± SD)

	Retrospective Cohort	Prospective Cohort (Consented)	P Value*
Preoperative Quick DASH	40 (23)	40 (19)	.86
Postoperative Quick DASH	27 (25)	18 (13)	.41

^*Determined by the Mann-Whitney test.

Next, we compared the preconsent and postconsent Quick DASH results within the prospective cohort. Study subjects did not alter their questionnaire responses after being informed of and consenting to participation in a study (39.0 + 17.8 prior to consent vs 9.7 + 19.1 after consent obtained, P = .98).

We analyzed the Quick DASH scores of the retrospective and prospective cohorts by insurance status. There were no statistically significant differences between the retrospective (not consented) and prospective (consented) cohorts either preoperatively or postoperatively for private insurance, Medicare, Medicaid, or workers’ compensation.

DISCUSSION

Our results do not indicate that the informed consent process introduced a systemic bias. The demographic characteristics, electrodiagnostic findings, carpal tunnel severity, and symptom duration were all similar between the retrospective and prospective cohorts. Quick DASH responses did not vary between the retrospective and prospective cohort or within the prospective cohort using the intake questionnaire responses as an internal control. Although there was a numerical difference of 9 points between the retrospective and prospective groups postoperatively, this finding was neither statistically significant (P = .41) nor clinically meaningful based on the minimal clinically important difference of 10 points. It is possible, however, that although we powered the study to identify a 10-point difference in Quick DASH score, the study was underpowered to identify small differences and would have identified a statistically significant small difference if the cohort were larger. Regardless of statistical significance, a difference of this magnitude would not demonstrate clinically meaningful bias. Alternatively, one could attribute the difference in score to measured confounding factors such as insurance status, or other unmeasured confounding factors.

Our results are most comparable to those of Fernald et al,⁹ who did not identify a Hawthorne effect in their study of patients with skin and soft tissue infections. McCarney et al⁸ found that patients in a dementia study with more intensive follow-up had significantly higher cognition scores than those with minimal follow-up, which suggests that awareness of participation in a study may overstate treatment benefit. Our findings, though not statistically significant, may also represent an overestimation of treatment effect in the consented cohort compared with the retrospective (not consented) cohort.

This study has several strengths. We performed a power analysis based on the minimal clinically important difference for Quick DASH and exceeded the number needed to identify a clinically meaningful difference in questionnaire responses. In addition, we were able to answer our question in 2 different ways. First, we compared a retrospective (unconsented) cohort with a prospectively consented cohort. Second, we used the prospective cohort as an internal control by comparing their intake questionnaire responses with responses after obtaining informed consent. We believe that, if a clinically meaningful consent bias had been present, this study would have identified such bias.

Although the aim of our study was not specifically to determine outcomes of patients after carpal tunnel surgery, the improvement in score reflects that of other carpal tunnel syndrome outcomes studies.^12,19–21 Overall, privately insured patients appeared to have better preoperative and postoperative Quick DASH scores than Medicaid, workers’ compensation, or Medicare patients. Because this was not the focus of our study, however, we did not carry out statistical analysis for this question.

We chose the Quick DASH form as the standard patient intake questionnaire in our practice because it is validated for a variety of conditions in the upper extremity, including carpal tunnel syndrome. Whereas other questionnaires are more responsive to detecting changes in carpal tunnel syndrome (for instance, Brigham and Women’s carpal tunnel questionnaire, Michigan Hand Questionnaire, and Carpal Tunnel Questionnaire),^11–13 we think that the Quick DASH score is an appropriate outcomes questionnaire for a general upper extremity practice. Given the study design, using a different questionnaire for the study would be impractical.

We chose carpal tunnel syndrome as the model disease for our study because it is common; several validated outcome measures are reported in the literature, including DASH; and a standard objective measure of the disease was available in the form of electrodiagnostic studies. We chose our minimum follow-up duration to be 6 weeks because most patients who undergo uncomplicated carpal tunnel release in our practice do not return for further follow-up. Using this time point ensured optimal participation in the study.

The study has several weaknesses. The ideal study design to answer our question would involve a randomized controlled trial in which an unconsented cohort is prospectively studied. However, institutional review boards are unlikely to approve such a study design. Next, although we desired follow-up at a single time point, for practical reasons the follow-up duration was variable, because of variation in patient follow-up preferences and compliance with completing the follow-up questionnaire. Second, the variability in timing of intake questionnaire completion (either at the first visit or on the day of surgery) introduces the possibility that patients who completed both preoperative questionnaires on the same day could recall their responses to the initial questionnaire. However, we think that if recall bias had a noteworthy influence on our findings, the first comparison (retrospective vs prospective) would have shown a difference in parameters, and the internal control comparison would have been without difference. Both of these analyses failed to support the idea that recall bias was a significant factor in our study. Moreover, we had a 100% consent rate for our study. Therefore, our study could not address the question of whether any identifiable patient factors that vary between consenters and nonconsenters (ie, selection bias) alter questionnaire responses.

Our findings provide support for comparable generalizability of studies using both an opt-in and opt-out approach to study patient recruitment, as long as the consent rate is high when an opt-in approach is used.