Short abstract
Overly strict interpretation of the law is hampering epidemiological research. Here, one research team shows why regulators and organisations holding data should adjust their approach
The United Kingdom's Data Protection Act 1998 has had a substantial impact on health research, although that was not its primary purpose. It is a wide ranging piece of legislation that safeguards individuals' fundamental right to privacy when personal data are processed.1 The act supplements the common law of confidentiality developed through judicial decisions. The medical and academic community is concerned that current interpretation of the law is changing the face of medical research.2 We use our experiences in studying UK military personnel to present empirical data in support of a more balanced interpretation of the act.
Data Protection Act and the law of confidentiality
Although some medical researchers blame the law for increasing the bureaucratic barriers to research,3 others, including the Information Commissioner and the Lord Chancellor, dispute this.4 The Lord Chancellor stated in the House of Lords:
At present the 1998 Act allows medical data to be used for any medical research purpose without the need for consent of individuals. It is not necessary to define the term “medical research” nor to make specific provision for it to include the monitoring of public health, which for these purposes is regarded as medical research.4
Current difficulties seem to stem from interpretations of the law that fail to appreciate the methods of epidemiology and the relatively minor privacy interferences that most epidemiological research entails.
Currently most codes of conduct recommend informed consent for any medical research, whether it involves direct contact with participants or access to their records. But as a matter of law, this is not an absolute rule. Both the law of confidentiality and the Data Protection Act envisage circumstances in which personal health information may be accessed and used for medical research without explicit consent or full anonymisation. Since the Human Rights Act 1998 became law, privacy interferences of this kind are permissible if the research investigates an important question, the research is in the public interest, is of a public nature (a term undefined in the Data Protection Act), and the degree of interference is proportionate to this goal (and no more than necessary).
Figure 1.
Statistics: missing in action
Credit: TOY/REX
Although the courts have not given an authoritative statement related to medical research, previous judgments suggest they would interpret current law as supporting large epidemiological studies that require record linkage, access to cancer registries, or data on names and addresses in order to identify potential participants (who would then be contacted in order to obtain informed consent for participation).5-7
Confusion also arises about the Data Protection Act's requirement that individuals be given information about the purposes of the proposed data processing.1 This is different from consent—it is the “fair processing requirement” based on a principle of “no surprises.” Even here the law is not as absolute as many believe. An epidemiologist who proposes to process personal data obtained from someone other than the patient must take proportionate steps to contact the patient as soon as practical to inform them of the processing. Disproportionately burdensome steps need not be taken, and the researcher is not obliged to wait for a response from the individual. Furthermore, when the research is historical or statistical, the fair processing requirement is relaxed, provided that the data are not used to take any decision relevant to that particular individual, that subsequent publication does not lead to identification of the subject, and that it is unlikely to cause substantial damage or distress.1
Since 2001, a further system has been in place in England (but not Scotland) whereby researchers can apply for permission to process health data without consent. The Patient Information Advisory Group advises the secretary of state for health whether a breach of confidentiality should be permitted for a necessary and proportionate interference. This is referred to as section 60 exemption.8
In theory, therefore, several avenues permit the proper use of personal information without always seeking informed consent. Despite this, many of those who control access to healthcare data are not allowing these legitimate and sensible exceptions to be put into practice. As an example, we describe the difficulties encountered by our research group.
The problem
Since 1995, our team has conducted several large epidemiological studies of military and former military personnel in the United Kingdom.9-11 Our original study, examining the health effects of service in the 1991 Gulf war, included 8195 participants.9 In this study, and in a subsequent larger study of personnel who participated in the 2003 Iraq war, we made initial contact using details provided by the Ministry of Defence (without their consent). The ministry released the data after taking into account the public health importance of the study and legal advice on the obligations of the previous Data Protection Act.
To compare the outcomes of those serving in the 1991 Gulf war with general military health we also collected data on the health of UK military personnel who served in UN peacekeeping operations in Bosnia in 1992-6.12 After we started these studies, concerns were voiced in the media and the veterans' community about possible cancer risks among Bosnia veterans, fuelled in part by speculation regarding depleted uranium exposure. As we had already been able to link records on Gulf war veterans with cancer registrations without explicit consent,13 we assumed that we would also be able to flag our Bosnia cohort with the NHS Central Register (which holds details of all cancer and death registrations in England and Wales).
However, when we attempted this in 2004 we were told that we needed section 60 exemption from the Office of National Statistics advisory group for medical research. After about eight months' delay we were told that participant consent was required in order for this sample to be flagged. Although the Office for National Statistics could give us details of those who had died and provide information to allow us to trace those who were alive to ask for consent, they were unable to grant us access to information on cancer diagnosis without consent.
The requirement to seek consent will introduce participation bias as seeking consent will inevitably reduce the response rate and those most likely to respond are likely to be those for whom the study has greatest salience—that is, those who have developed cancer.
As a result, the study is likely to become too small and biased to reach useful conclusions. We believe that the Office of National Statistics has not to date taken sufficient account of the exceptions permissible in law outlined above. We argue that the disclosure that we are requesting is indeed proportionate, and unlikely to cause distress, and that the information cannot be obtained by any other means. Discussion with ONS is ongoing, but until now we had no empirical data to support or refute our position. Other researchers face similar predicaments. We therefore examined our previous data for evidence, which we present below.
Response rates vary by method of contact
Epidemiological research requires representative samples and high response rates. Response rates matter for two reasons. Firstly, if the sample size is reduced, the study loses statistical power and, therefore, may not be able to identify (and quantify) any true effects. Secondly, a low response rate means that participation bias is almost inevitable. Epidemiologists therefore spend a large proportion of their time selecting an appropriate sample and then contacting and tracing them to get acceptable response rates.14,15 One way of avoiding disclosing personal information to researchers is to use proxies to contact subjects on behalf of the research team. The current data controller—for example, the individual's general practitioner or employer—contacts the potential subject and invites him or her to contact the researcher.
Our experience is that using proxies to obtain consent is unsatisfactory. In the first place it is difficult to secure the necessary commitment from proxies because of the resource implications. More importantly, our data show a hierarchy of success for various methods of contacting people (table 1). Potential participants have greater trust in the researchers, especially when they meet them face to face, than they have in their employers.
Table 1.
Response rates (%) for military studies by method of contact, reasons for non-response, and rate of complaints
| Method of contact | Response rate (%) | Refusal (%) | Wrong address or number (%) | Person not available (%) | Reason unknown (%) | Complaints (%) |
|---|---|---|---|---|---|---|
| Face to face contact researcher | ||||||
| MCTC* (n=135) | 134 (99.3) | 1 (0.7) | NA | NA | NA | 0 (0) |
| Telic† (n=6258) | 3088 (49.3) | 26 (0.4) | NA | 3144 (50.2) | NA | 1 (0.02) |
| Telephone contact by researcher‡ | ||||||
| Gulf 19 (n=200) | 139 (69.5) | 22 (11) | 11 (5.5) | NA | 28 (14.0) | 2 (1) |
| Veterans17 (n=496) | 315 (63.5) | 56 (11.3) | 125 (25.2) | NA | NA | 2 (0.4) |
| Telic† (n=675) | 61 (9.0) | 34 (5.0) | 514 (76.1) | 66 (9.8) | NA | 1 (0.1) |
| Postal questionnaire sent by researcher | ||||||
| Gulf 1 (1st wave) (n=11 710)9 | 4511 (38.5) | 7 (0.06) | 418 (3.6) | NA | 6774 (57.8) | 2 (0.02) |
| Gulf 1 (2nd wave) (n=7192)9 | 1684 (23.4) | 163 (2.3) | 401 (5.6) | NA | 4194 (68.7) | 0 (0) |
| Gulf 1 (3rd wave) (n=5345)9 | 1308 (24.5) | 315 (5.9) | 393 (7.4) | NA | 3329 (62.3) | 0 (0) |
| Telic† (n=15 895) | 5526 (34.8) | 21 (0.1) | 880 (5.5) | NA | 9468 (59.6) | 4 (0.03) |
| Contact by government departments | ||||||
| Veterans17§ (n=65) | 8 (12.3) | NA | 10 (15.4) | NA | 47 (72.3) | NA |
| Administrative discharges*¶ (n=6) | 0 | NA | 5 (83.3) | 1 (16.7) | NA | NA |
Study of post discharge mentoring of vulnerable service leavers, the Military Correction and Training Centre Colchester and those who had received an administrative discharge (in progress).
Study of the physical and psychological health of those who deployed on Operation Telic (in progress). The study involves postal questionnaires being sent to participants and base visits where participants are invited to fill in their questionnaires with researchers to hand for advice.
Studies (or parts of studies) where initial contact from the research team is established on the telephone. For the Gulf 1 study and the Telic study, this was attempted telephone contact with people who had not returned the postal questionnaire. For the veterans' study, it was telephone contact with individuals who had taken part in a previous study and had consented to follow-up. The low response rate in the Telic study reflects the fact that tracing is ongoing.
The Department of Work and Pensions offered to contact by letter (on our behalf) those whose contact details we were unable to trace.
The Ministry of Defence offered to contact service leavers on our behalf. They were able to reach only one individual who had been given an administrative discharge, and he was unavailable at the time of the call. This methodology was subsequently abandoned.
In our studies, we obtained the highest response rate from face-to-face interviews conducted by civilian researchers within a military prison. The interviews were entirely voluntary, and all participants were given an information sheet and completed a consent form. The participants in this study ought to be the sample hardest to recruit, being largely young men with high rates of substance misuse and antisocial behaviours. This suggests that participants' willingness to consent is influenced by who does the asking.
Table 1 also supports the view that few people complain about being contacted by a researcher. Taken together, these data support the argument that it is both necessary and proportionate to grant researchers access to personal data to enable them to contact people for consent and that the use of proxies is not a satisfactory alternative.
True refusal rates are generally low
One objection to a more liberal interpretation of the Data Protection Act is that low response rates in epidemiological studies reflect an informed decision not to take part. If this is so, these preferences should be respected. But is it? In our experience, low response rates do not stem from people objecting to the research topic or protocol (table 1). Instead non-response is often due to more mundane problems of tracing (when contact details are out of date) or people's failure to complete the questionnaire (apathy). This is evidence that the privacy interference arising from epidemiological research without consent is not as serious as sometimes presumed.
Non-response is not necessarily due to distress
Because we have been better resourced and more persistent than is possible for some follow up studies, we have been able to look at reasons for non-response in some detail at each wave of data collection and after different methods of tracing or outreach. We have consistently found, as others have,16 that non-response is related to demographic factors, and that non-responders are more likely to be young, unmarried men.9,11,17 We have not found that lack of response is related to health factors, such as greater emotional distress from the study; in fact, we have reported the opposite.18 Non-response is therefore more likely to be due to factors such as time constraints (for long postal questionnaires) or lack of interest than to distress. This again speaks to the proportionality of some privacy interferences—that is, research where it is reasonable to think that a poor response rate is due to apathy rather than principled objection.
Problem of natural attrition
At the point of recruitment to a cohort study, it is not always possible to gain consent for what will later turn out to be important research questions. It is never possible to predict all relevant exposures that might happen during active service—information about some environmental exposures may not appear until many years later. In our example, concern about possible health effects of depleted uranium used in former Yugoslavia and the Gulf war took some years to develop, by which time most of those potentially exposed had left the military. They could not be easily contacted by either their former employers or the research team since last known addresses are valid for only a brief period in most service leavers.
Currently, for studies of such late occurring effects data controllers and regulators prefer that the cohort is re-contacted. But how possible or proportionate is this? Table 2 shows the natural attrition over time for our Gulf cohort, whom we contacted in 1997,9 2001,11 and 2003,17 and concurrent decline in the representativeness of the cohort. This bias is important because restricting the analysis to those whom we can trace increases the possibility of missing an important health effect.14 The legal implication is that, in longitudinal cohorts of this kind, it is sometimes necessary to proceed without specific consent. There is also a case for arguing that it would be proportionate to do so as explained below.
Table 2.
Attrition of Gulf war cohort over time
Research participants trust epidemiologists, once engaged
Regulators seem to presume that participants distrust epidemiologists. Hence they take the view that it is a serious breach of an individual's autonomy and privacy to proceed without specific consent for subsequent research. Our data do not support this conclusion. When we eventually manage to contact individuals for follow-up studies, few refuse access to further data such as medical or vaccine records (table 3). Our data suggest that study participants, once engaged, continue to trust medical researchers with personal health data.19 From this, it is reasonable to conclude that if we were able to locate the missing members of the cohort, they would be likely to have similar views on permitting access to information. Thus, allowing the research to proceed without follow-up consent (that is, allowing it to proceed on the basis of the initial consent) would not be a disproportionate interference in privacy.
Table 3.
Numbers (percentages) of participants consenting to release of additional information
| Study | Medical records (%) | Vaccination records (%) | Next of kin |
|---|---|---|---|
| Gulf 19 (n=8195) | 7785 (95.0) | NA | NA |
| Telic* (n=7535) | 7160 (95.0) | 7258 (96.3) | 6248 (82.9) |
| MCTC† (n=135) | 135 (100) | NA | 125 (93)‡ |
| Screening study§ (n=2925) | 2182 (74.6) | NA | NA |
Study of the physical and psychological health of those who deployed on Operation Telic (in progress).
Study of post discharge mentoring of vulnerable service leavers, the Military Correction and Training Centre Colchester. We sought consent for results of educational tests taken during their prison stay rather than for medical records.
Those who did not provide data often did not consider themselves to have a next of kin or could not remember their full address.
Screening of active service personnel for mental health problems: 18.5% of the sample refused and 6.8% did not fill in this part of the consent form.
Risks of participation are negligible
Governing bodies often presume that information based research requires the same review procedures and strict principles about consent as interventional research. Little regard is given to the different sorts of risks entailed. Informational privacy is important, but it is not nearly as fundamental as the right not to be physically assaulted against your will by a medical researcher, which is the issue potentially at stake with an intervention study. The proportionality principle is misapplied if the risks associated with epidemiological research are equated with interventional research.
Existing research about research participants' attitudes tend to support the view that the risks of epidemiological surveys, even when direct contact is made, are low.20 In our studies few people complain about taking part, as shown in table 1, and the objective rate of side effects or adverse consequences is extremely low. To our knowledge, our studies have caused distress on just two occasions in the past 10 years, and we have contacted more than 25 000 people in that time. On both occasions we inadvertently wrote to the family of a serviceman who had just died. This was regrettable, but none of the proposed alternatives (such as requiring a proxy to make contact on our behalf) would have avoided this. We have reduced this risk by improving the way in which our data are kept contemporaneous. More cumbersome methods of contact will increase (not decrease) this problem, by building in a time lag. Many of the study participants we directly contacted indicate that they appreciate the time we spend with them and our ability to point them towards services where relevant.
These data should encourage a less rigid policy towards data sharing in epidemiological research. Epidemiological research is usually a positive or at worst neutral experience for participants, and deviations from the consent rule should more often be perceived as proportionate.
Summary points
The UK Data Protection Act (1998) is changing the way that medical research is conducted
The law allows personal information to be used and disclosed without explicit consent, subject to certain safeguards, when it is impractical to obtain consent and an important public interest is at stake
Despite this, some data controllers continue to interpret the law in a restrictive way
The main barriers to epidemiological research are that people cannot be contacted or are approached by proxies who know little about the study
Adverse events or detriment from participation in this type of research are extremely rare
Conclusions
The current confused, and confusing, legal position is acting as a barrier to important medical research. Nothing proposed in this article is against either the spirit or the letter of the legal framework. The law has foreseen the importance of data sharing, the difficulties with anonymising data fully, and the ethical reasons for balancing the right to privacy against other rights and interests. But those who implement the law have not always grasped this. Accordingly, they fail to recognise situations where it is necessary and proportionate to relax consent requirements in epidemiological research.
We are not arguing that epidemiological research should always proceed without consent. But it should be allowed to do so when the privacy interference is proportionate. Regulators and researchers need to improve their ability to recognise these situations. Our data indicate a propensity to over-predict participants' distress and under-predict the problems of using proxies in place of researchers. Rectifying these points would be a big step in the right direction.
Contributors and sources: SW is a member of the working party of the Academy of Medical Sciences on the Use of Patient Data in Medical Research, has a longstanding interest in this subject, and conceived the idea for the study. AI extracted the relevant data from previous studies, did the analysis with assistance from NF, and co-authored the paper with SW. KL provided legal advice. MH and SW led the original research team and coordinated the series of studies described, and both commented further on the paper. MH dealt with AGMR/ONS.
Competing interests: None declared.
References
- 1.Data Protection Act 1998. Section 33(2). www.uk-legislation.hmso.gov.uk/acts/acts1998/19980029.htm (accessed 15 Sep 2005).
- 2.Coleman MP, Evans BG, Barrett G. Confidentiality and the public interest in medical research—will we ever get it right? Clin Med 2003;3: 219-28. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Strobl J, Cave E, Walley T. Data protection legislation: interpretation and barriers to research. BMJ 2000;321: 890-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Falconer C. House of Lords Hansard 2000 Nov 14. www.parliament.the-stationery-office.co.uk/pa/ld199900/ldhansrd/vo001114/text/01114-35.htm (accessed 15 Sep 2005).
- 5.Campbell v Mirror Group Newspapers Ltd [2004] 2 AC 457 (HL).
- 6.R v Chief Constable of South Yorkshire ex parte S and Marper [2004] 4 All ER 193 (HL).
- 7.R v Department of Health Ex p Source Informatics Ltd [2001] QB 424 (CA).
- 8.Health and Social Care Act 2001. London: Stationery Office, 2001. www.uk-legislation.hmso.gov.uk/acts/acts2001/20010015.htm (accessed 3 Jan 2006).
- 9.Unwin C, Blatchley N, Coker W, Ferry S, Hotopf M, Hull L, et al. The health of United Kingdom Servicemen who served in the Persian Gulf war. Lancet 1999;353: 169-78. [DOI] [PubMed] [Google Scholar]
- 10.Ismail K, Kent K, Brugha T, Hotopf M, Hull L, Seed P, et al. The mental health of UK Gulf war veterans: phase 2 of a two phase cohort study. BMJ 2002;325: 576-81. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Hotopf M, David AS, Hull L, Nikolaou V, Unwin C, Wessely S. Gulf war illness—better, worse, or just the same? BMJ 2003;327: 1370-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Hotopf M, David AS, Hull L, Ismail K, Palmer I, Unwin C, et al. The health effects of peace-keeping in the UK armed forces: Bosnia 1992-1996. Predictors of psychological symptoms. Psychol Med 2003;33: 155-62. [DOI] [PubMed] [Google Scholar]
- 13.Macfarlane GJ, Biggs AM, Maconochie N, Hotopf M, Doyle P, Lunt M. Incidence of cancer among UK Gulf war veterans: cohort study. BMJ 2003;327: 1373. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Jousilahti P, Salomaa V, Kuulasmaa K, Niemela M, Vartiainen E. Total and cause specific mortality among participants and non-participants of population based health surveys: a comprehensive follow up of 54 372 Finnish men and women. J Epidemiol Community Health 2005;59: 310-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Prince M, Stewart R, Ford T, Hotopf M. Practical psychiatric epidemiology. Oxford: Oxford University Press, 2003.
- 16.Eaker S, Bergstron R, Bergstrom A, Adami H, Nyren O. Response rate to mailed epidemiologic questionnaires: a population-based randomized trial of variations in design and mailing routines. Am J Epidemiol 1998;147: 74-82. [DOI] [PubMed] [Google Scholar]
- 17.Iversen A, Dyson C, Smith N, Greenberg N. Goodbye and good luck; the mental health needs and treatment experiences of UK veterans. Br J Psych 2005;186: 480-7. [DOI] [PubMed] [Google Scholar]
- 18.Wood AM, White IR, Hotopf M. Using number of failed contact attempts to adjust for non-ignorable non-response. J R Stat Soc (in press).
- 19.Muir-Gray JA. Post modern medicine. Lancet 1999;354: 1550-3. [DOI] [PubMed] [Google Scholar]
- 20.Tyrer P, Seivewright H, Ferguson B, Johnson T. “Cold calling” in psychiatric follow up studies: is it justified? J Med Ethics 2003;29: 238-42. [DOI] [PMC free article] [PubMed] [Google Scholar]