HIPAA Authorization and Survey Nonresponse Bias

Abstract

Objectives

To extend earlier work¹ that demonstrated that a HIPAA authorization form (HAF) introduced potential nonresponse bias (toward healthier respondents).

Research Design

The sample frame from the earlier experiment was linked to administrative medical record data enabling the comparison of background and clinical characteristics of each set of respondents (HAF and No HAF) to the sample frame.

Subjects

6,939 individuals residing in Olmsted County, Minnesota who were mailed a survey in September 2005 assessing recent gastrointestinal symptoms with an embedded HAF experiment comprise the study population.

Measures

The outcomes of interest were response status (survey returned vs. not) by HAF condition (randomized to receive HAF or not). Sociodemographic indicators included gender, age, and race. Health status was measured using the severity weighted Charlson Score and utilization was measured using ER visits, hospital admissions, clinic office visits, and procedures.

Results

Younger and nonwhite residents were under-represented and those with more clinical office visits were over-represented in both conditions. Those responding to the survey in the HAF condition were significantly more likely to be in poor health compared to the population (27.3% with 2+ comorbidities vs. 24.6%, p=0.02).

Conclusions

The HAF did not influence the demographic composition of the respondents. However, counter to earlier findings based on self-reported health status¹, responders in the HAF condition were slightly sicker than in the non-HAF condition. The HAF may introduce a small amount of measurement error by suppressing reports of poor health. Further, researchers should consider the impact of the HAF on resultant precision, respondent burden, and available financial resources.

INTRODUCTION

Surveys will likely play a key role in health research, providing information on health and health care for large numbers of people in a cost effective manner. Consequently, it is important to constantly gauge what might or might not adversely effect health survey participation. A key provision in the Health Insurance Portability and Accountability Act (HIPAA) of 1996 is that those that use or disclose protected health information can only do so with individual authorization unless that requirement is formally waived by an Institutional Review Board. There are elements of HIPAA authorization that may adversely impact prospective respondents’ willingness to participate in health surveys. First, the language in the HIPAA Authorization Form (HAF) is thought to erroneously convey the notion that by signing the form, prospective respondents’ protected health information will no longer be protected². This may elicit concerns over privacy that may, in turn, lessen the likelihood of response to the survey request^3–6. Second, many HAFs require participants’ signatures which have also been shown to negatively affect willingness to participate^5–7. The evidence investigating the effects of inclusion of the HAF on survey participation is equivocal with some studies showing no effect of the HAF on participation^{8, 9} and some showing a rather large negative effect of the HAF on participation ^{10, 11}. In our own research in this area – reported recently in Medical Care¹ – we found that inclusion of even a minimally burdensome version of the HAF (1 page) reduced survey response rates by up to 15 percentage points.

In our earlier study, two observations, coupled with limitations in our data at the time, brought about the need for further investigation. First, even though we observed a 15 percentage point difference in the response rates between those who did and did not receive the HAF (39.8 and 55.0 percent respectively), we found little difference in the sociodemographic profile of respondents and nonrespondents in terms of age, gender, race/ethnicity, and education. However, our method of comparing survey respondents and nonrespondents using estimates for the population from the U.S. Census, was less than ideal. This technique is the nonresponse bias analysis tool most often used by survey researchers and others¹² but is limited by the possibility that the question-asking and method of data collection used by the Census may differ from the those used in the survey to which the population parameters are being compared. Such differences might introduce error that may artificially over- or under-estimate concordance and therefore bias. A preferred method is to match sample respondents to an external database at the individual level because it limits the error introduced by the aforementioned sources. We have done just that and report on the results herein. Evidence from recent systematic reviews of the literature suggesting that low response rates do not necessarily portend response bias (or the converse), ^{12, 13} underscores the importance of extending our work in this area.

Because similarity or differences between respondents and nonrespondents on a limited set of sociodemographic characteristics does not necessarily translate to comparable similarities or differences in survey responses^14–18, we assessed whether there was inconsistency in the responses to key outcome variables between the two experimental conditions in our 2007 study. Specifically, we selected general health, the mental health and physical health scales of the SF-12¹⁹, the number of abdominal symptoms experienced, Body Mass Index (BMI), smoking status, and past 30 day drinking of alcohol as the key outcome variables in our analysis. We found significantly (p ≤ 0.01) higher proportions self-reporting general health and non-smoking in the HAF condition than in the No HAF condition. Whether this finding was due to the HAF influencing the self-selection of a slightly healthier population into the responding sample or suppression of their reports of ill health due to privacy or other concerns could not be ascertained with the data available in our 2007 HAF experiment study. This is the second major issue that we hope to resolve with linked data in the current paper.

This paper reports a systematic analysis of survey nonresponse bias using linked data collected from two sources. The first source is the aforementioned survey-based experiment evaluating the effect of the HAF on response rates and nonreponse bias¹. The second source of data is the Rochester Epidemiology Project (REP), the medical record linkage system for health care providers to residents of Olmsted County²⁰. This study used an unprecedented amount of health-related information on both respondents and nonrespondents toward the goal of determining how nonresponse bias might be affected by the inclusion of a HAF and its signature requirement. The key research question to be addressed in the analysis was: Does including a HAF impact the participation of different types of Olmsted County residents? Specifically, does including a HAF impact the participation of individuals of certain health statuses and characterized by different health care seeking behavior?

METHODS

The HIPAA Authorization Form (HAF) Experiment Study

The data on response status (respondent vs. nonrespondent) come from a sequential mixed mode, mail and telephone survey conducted between September 2005 and April 2006 by the Mayo Clinic Survey Research Center. The population for this survey included non-institutionalized residents of Olmsted County, Minnesota aged 18 and older as identified in a purchased list-based sample of Olmsted County residents. The survey instrument included items required for assessing recent (past 3 months) gastrointestinal symptoms, current height and weight, weight loss attempts, eating behavior, physical activity and exercise, presence of diabetes, familial gastrointestinal symptoms, tobacco and alcohol consumption, and selected demographics. Further details of the methodology are available elsewhere ¹.

The 6,939 eligible cases were randomly assigned to receive either a mailed survey packet with two copies of a one-page HIPAA Authorization Form (HAF) (one to sign and send in, another to keep for their records) or a mailed survey packet without the HAFs. We received a special exception from the Mayo Clinic Institutional Review Board (IRB) to withhold the HAF from half the sample purely for purposes of this experiment. The content of the HAF was consistent with current HIPAA guidelines (http://www.hhs.gov/ocr/hipaa).

Initial non-responders were sent a second survey 3 weeks later. For those assigned to the HAF condition another two copies of the HAF were included in the second mailing. For those that did not respond to either mailing, a telephone interview was attempted approximately 5 weeks after the initial mailing. When a phone interview was completed, the interviewer indicated that in order to use the supplied interview data, the respondent had to fill out the HAF. If the respondent did not have a copy of the HAF available from the prior mailing, another one was offered and sent to them the day of the interview. The 6,939 individuals who were randomized to the two conditions, regardless of respondent status, comprise the study population.

The Rochester Epidemiology Project (REP)

The sampling frame for the HAF Experiment Study was linked to the Rochester Epidemiology Project (REP). Each health care provider in Olmsted County (home of Mayo Clinic, Olmsted Medical Center, and the Rochester Family Medicine Clinic) uses a unit (or dossier) medical record system whereby all data collected on an individual are assembled in one place. Through the REP, these health care providers have agreed to share their patient records for research studies approved by the Mayo Clinic and Olmsted Medical Center Institutional Review Boards. Each medical care site that participates in the REP also solicits and documents permission from individual patients for their records to be used in research studies. Currently, 95% of patients have granted permission for their records to be used in research. In December, 2008, the REP contained 1,145,856 patient records from 49 different health care providers that matched to 486,564 individual patients who had been residents of Olmsted County at some point between January 1, 1966 and December 31, 2008. The diagnoses assigned at each visit are coded and indexed electronically. In addition to medical diagnoses, hospital admissions and surgical or non-surgical procedures, the REP includes demographic information such as age, gender, and race.

Data Linking Process

For the current study, individual health care data from the REP were merged with data from the HIPAA Authorization Form (HAF) Survey Experiment. We matched the data in the sample list against the entire REP database to determine what proportion of our sample had available health care information. Overall, we were able to match almost 97% of the cases in the sample file to members in the REP database. Individuals who were not matched to anyone in the REP may not truly be Olmsted County residents, or, alternatively, may not have yet received medical care in Olmsted County. Primary analyses focused on the 6716 individuals for whom health care information was available. This study was reviewed and approved by the IRBs at both Mayo Clinic and Olmsted Medical Center.

Measures

Measures from the HAF Experiment Study

The survey instrument utilized in the HAF Experiment Study included items ranging from gastrointestinal symptoms to tobacco and alcohol consumption along with a range of demographics. Here, we are interested only in whether the administrative records differed by response status across the two experimental conditions (No HAF versus HAF). Response status was operationalized to include those who completed a mailed survey or telephone interview (at least two-thirds of the items completed) and within the HAF condition, returned the signed HAF. In other words, to be considered a response in the HAF condition, respondents had to both complete the survey and send in a signed HAF. Refusals, noncontacts, and non-signers constituted the non-response level.

REP-based Measures

Selected demographic variables were obtained from the REP including birth date (age), gender, and race/ethnicity. All medical and surgical diagnoses received by patients at a health care site participating in the REP are coded using either Hospital Adaptation of the International Classification of Diseases (HICDA)²¹ or the International Classification of Diseases, 9^th Edition (ICD-9) codes, and are catalogued and stored in the REP medical and surgical indexes. These diagnostic codes, with associated dates, are electronically retrievable for a given list of individuals. An automated macro (computer instruction) that calculates the severity-weighted Charlson index^{22, 23} based on these diagnoses was used to provide a summary score of comorbidity based on diagnoses over the past decade. The surgical index was also searched to ascertain whether each subject had ever had a surgical or non-surgical procedure at one of the hospitals in Olmsted County from 1995–2006. Finally, the number of emergency room visits, outpatient visits, and hospital admissions from 2005–2006 were calculated for each study subject.

Statistical Analysis

For each of the background and clinical variables, cut-offs were chosen to facilitate analysis and interpretation, informed by a review of the marginal distributions at the item level to identify natural breaks, and designed to accord with prior authorization studies undertaken in Olmsted county using the REP²⁴. These variables are as follows: gender, age (18 to 34, 35 to 49, 50 to 64, 65+), race (Black/African American, White, Other), weighted Charlson score (2 or more vs. <2), emergency room (ER) admissions and hospital admissions (yes vs. no), clinic office visits (3 or more vs. <3), and procedure (≥1 year vs No Procedure). In sensitivity analyses, data were re-analyzed to assess the appropriateness of the cut-offs. In addition, variables were treated as continuous in some analyses, and both parametric and nonparametric methods were examined.

To test for the presence of nonresponse bias in each experimental condition using medical record and administrative data from the REP rather than from Census population estimates as we had done in our prior work, we compared the gender, age, race, Charlson commorbity scores, emergency room admissions, hospital admissions, clinic office visits, and medical procedures to those characteristics in the entire linked data set (n = 6,716) using chi-square goodness-of-fit tests with the sample frame distributions as the population estimates. We also formally tested for differences between the Non-HIPAA and HIPAA conditions. For these analyses, chi-square tests were calculated. All analyses were performed using SAS v. 9.1 software²⁵. A p-value of < 0.05 was regarded as statistically significant.

RESULTS

As can be seen in Table 1, younger (≤50 years) and nonwhite residents were under-represented and those with higher levels of clinic office visits were over-represented in both the Non-HAF and HAF conditions compared to the population in the linked data set. The only clinical point of differentiation between the two experimental conditions was in the area of comorbities. Those responding to the survey in the HAF condition were significantly more likely to have a weighted Charlson score of 2 or more as compared to the population (27.3% vs. 24.6%, p=0.02), indicating that a slightly sicker population responded to the survey with the HAF. No significant differences between the Non-HIPAA and HIPAA conditions were observed (data not shown).

Table 1.

Distribution of Selected Sociodemographic and Clinical Variables for the Total Linked Data Population and Survey Respondents by Experimental Condition

Variable	Population (n=6716)	Non-HIPAA Response (n=1863)	HIPAA Responsea (n=1357)
Background
Gender (%)
Female	47.4	48.4	49.9
Male	52.6	51.6	50.1
P-value		0.41	0.07
Age (%)
18 to 34	16.8	14.5	13.3
35 to 49	32.8	30.2	31.5
50 to 64	28.1	31.9	29.6
65 +	22.3	23.4	25.6
P-value		0.0002	0.0004
Race (%)
Black/African American	1.4	0.86	0.66
Other	11.8	8.9	9.6
White	86.9	90.2	89.8
P-value		<.0001	0.0022
Clinical
Charlson Score (severity weighted) (%)
< 2	75.4	75.8	72.7
2 or More	24.6	24.2	27.3
P-value		0.69	0.02
ER Admission in 2005 and 2006 (%)
No	69.8	71.2	71
Yes	30.2	28.8	29
P-value		0.18	0.35
Hospital Admission in 2005 and 2006 (%)
No	78.1	79.2	78
Yes	21.9	20.8	22
P-value		0.26	0.96
Clinic office visit in 2005 and 2006 (%)
< 3	37.6	31.9	30.4
3 or more	62.4	68.1	69.6
P-value		<0.0001	<0.0001
Procedure (%)
No Procedure	42.3	39.1	39.1
≥1 year	57.7	60.9	60.9
P-value		0.17	0.24

NOTE: Estimates compared each experimental group with characteristics from the entire population in the linked dataset using chi-square goodness-of-fit tests where the expected values are based on the population proportions. In separate analyses, no significant differences between the Non-HIPAA and HIPAA conditions were observed (data not shown).

^aTo be considered a “response” in this condition, respondents had to both complete a survey AND send in a signed HIPAA Authorization Form (HAF).

For the most part, our primary findings were not altered by the varied approaches. The only exception to this general trend was for the Charlson score where we found that the findings either reversed (viz. the distribution was different from the population in the non-HIPAA condition) when we collapsed Charlson scores into three levels (0, 1 or 2, > 2 OR 0, 1, 2+) or slipped into statistical non-significance altogether when we treated Charlson score as continuous.

DISCUSSION

Overall, we found that the sociodemographic profile across our two experimental conditions (HAF vs. No HAF) did not differ. However, both groups of responders did differ from the population as a whole with respect to age and race. This finding is consistent with what we observed in our 2007 Medical Care article ¹ where we compared respondents across conditions to Olmsted County population controls obtained via U.S. Census parameters. This finding offers tacit support to the common practice of comparing the characteristics of survey respondents to similar estimates from other sources such as the U.S. Census as a method of gauging the presence or absence of nonresponse bias. This practice is important to validate because when researchers have very little information on both respondents and nonrespondents at the individual level (which is often the case ¹²) it is the only method available for estimating nonresponse bias. However, external data sources such as those obtained from the U.S. Census often do not include information that is more proximal to health survey subject matter, such as medical diagnoses and health care utilization patterns. Using this type of unique data afforded by the linked REP information, we found that a slightly sicker population, in terms of the number of comorbidities, was more likely to respond to the survey AND sign a HAF than those completing a survey in the arm where they did not receive a HAF.

Our finding of sociodemographic equivalence between the No HAF and HAF conditions runs counter to what has been observed in the few studies investigating the effects of the HIPAA authorization on response bias. For example, Krousel-Wood and colleagues²⁶ found that written informed consent and HIPAA authorization resulted in lower participation among African Americans, females, and persons under 75 years of age in a cross sectional survey of older patients with hypertension. Bolcic-Janovic and colleagues found that among individuals that had been hospitalized in the past calendar year, men and older adults were more likely to return an authorization form as part of a phone survey ⁷. In a qualitative study investigating the effect of including a signed HIPAA authorization requirement on willingness to participate in a hypothetical clinical research study on antihypertensive medication, Dunlop and colleagues²⁷ found that males, those 40 years or older, and those with high school education or less were less likely to agree to study participation in the HIPAA authorization plus standard consent condition than those subjected to standard consent alone. Finally, in a telephone survey of patients with acute coronary syndrome where informed consent forms were mailed in advance of requesting signed permission to call, Armstrong and colleagues²⁸ found that patients who did not sign an authorization form tended to be younger, a member of a minority racial or ethnic group, and unmarried than those willing to do so.

In the realm of clinically-relevant differences between the No HAF and HAF conditions, our one significant finding of higher comorbidities in the group willing to complete the survey, again runs counter to what has been seen in the literature. In the Armstrong et al.²⁸ study mentioned above, those authorizing the subsequent telephone interview had significantly lower mortality rates at 6 months but no differences in myocardial infarction, stroke, or re-hospitalization between the 2 groups. If one construes lower 6 month mortality rates as indicative of greater health at the time of the survey request, the findings by Armstrong and colleagues²⁸ are at odds with our observation that those with poorer health at the time of the survey request were more likely to complete the survey with a signed HAF.

Why our findings differ from those observed in other studies is unclear but may be due to the fact that the literature in this area has focused on more homogeneous patient populations of various sizes. As indicated earlier, the Krousel-Wood et al.²⁶ study focused on a small (n = 177) sample of older patients with hypertension and the Dunlop et al.²⁷ study focused on a purposive sample of 384 African American patients from four metropolitan primary care clinics. The Armstrong et al.²⁸ study, while larger than the other two (n = 1221) still focused on patients with acute coronary syndrome. The Bolcic-Janovic study, while also larger (n=5,859), focused just on people that had been hospitalized for medical or surgical treatment in the past year ⁷. It may be that the findings from these smaller and/or specialized samples cannot be generalized to our larger (n = 6,716) sample of community residents.

The current findings also run counter to what we observed in our earlier report¹ where we found significantly higher proportions self-reporting general health and non-smoking in the HAF condition than in the No HAF condition. At first blush, this suggests that those willing to complete the survey and sign the HAF are healthier than those unwilling to do so; a finding similar to that observed by Armstrong et al.²⁸. However, our observation that those completing the survey and signing a HAF were less healthy (or, as shown in some of the sensitivity analyses, no different) in terms of the number of comoridities found in the REP administrative and medical record data raises a question about the validity of self-reported data. It is possible that the earlier finding might be due solely to suppression of self-reports of ill health in the HAF condition rather than differential selection and, as such, a case of measurement error rather than mere nonresponse error. Dunlop and colleagues²⁷ hypothesize that the viewing of the HAF itself may have differentially biased self-reports of willingness to participate in their hypothetical clinical research study. This finding highlights the limitations associated with relying solely on self-reports of health as indicators of response bias as past researchers have been prone to do, and underscores the importance of utilizing an external database that can characterize respondents and nonrespondents at the individual level as we have done in the current investigation. However, it is acknowledged that administrative and medical record data may also be prone to measurement error associated with such things as variability in data item definitions, data collection techniques, and cleaning processes over time ^{29, 30}. In addition, our findings relating to commorbidities varied greatly depending on how we treated the Charlson scores in our sensitivity analyses. It may be that our selection of the Charlson score as an indicator of health, and the manner in which we treated it analytically, may be driving our main results. However, the Charlson measure has been found to be an effective method of estimating future morbidity and mortality in longitudinal studies,²² underscoring its utility as a measure of current health. Nonetheless, future researchers should pursue further work on this topic.

In conclusion, the results of the present study demonstrate that the 15 percentage point reduction in the observed response rates brought about by the introduction of the HAF to the survey request did not portend systematic bias in the sample. This accords with emerging evidence suggesting only a weak relationship between a survey’s response rate and its response bias^{12, 13}. Furthermore, our findings do not imply that the inclusion of the HAF has a benign effect on health studies just because of our observed lack of nonresponse bias. First, we saw a response rate of 39.8% in the HAF condition and 55.0% in the non-HAF condition in our original study. Given our original sample of 6,939, had no one been sent the HAF we would estimate responses from a total of 3,816 and if all had been sent the HAF, 2,762. The inclusion of the HAF would therefore decrease our analytical sample by 1,054 individuals. This loss of sample is associated with real decreases in the relative precision of our estimates. For our survey estimate reported in the original study that approximately 10% of the population smoke, out margin of error would decrease from 1.1% to 0.9% with the larger sample. Similarly, for our reported estimate of BMI, we would be able to estimate the mean to within 0.190 units as compared to 0.225 units. Second, the additional study costs incurred as a result of increased printing and postage for the two HAFs in each mailing increases the overall cost per completion. While relatively minor on a per packet basis, these costs could be quite substantial in large scale studies such as ours. These additional costs could be quite burdensome as well for those facing strict financial constraints on even smaller studies. The larger lost cost is associated with the loss of information from costly telephone interviews that cannot be used due to the absence of a signed HAF. While the finding of lack of nonresponse bias with the inclusion of a HAF is good news for those required to include this form, its impacts are far from benign. There is real loss in statistical power which can translate into more expensive survey protocols to achieve the same level of confidence in one’s findings. Further work is needed in order to determine the best way to mitigate the loss of power associated with the HAF.