Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2010 Jan 1.
Published in final edited form as: Soc Sci Med. 2008 Nov 10;68(1):183–191. doi: 10.1016/j.socscimed.2008.10.001

Patients' perceptions of research in emergency settings: a study of survivors of Sudden Cardiac Death

Neal Dickert 1,, Nancy E Kass 2
PMCID: PMC2660168  NIHMSID: NIHMS89826  PMID: 19004536

Abstract

Conditions such as stroke, sudden cardiac death, and major traumatic injury are major causes of morbidity and mortality, and there is a need for clinical research to improve treatment for these conditions. However, because informed consent is often impossible, research in these situations poses ethical concerns. Despite growing literature on the ethics of emergency research, little is known about the views of relevant patient populations regarding research in emergency settings conducted under an exception from informed consent (EFIC).

In this qualitative study, survivors of sudden cardiac death (SCD)- recruited from an outpatient cardiology clinic in late 2005- were asked their views on scenarios representing different types of EFIC research. Patients were generally accepting of such research, more than previous studies would have predicted. Their concerns focused primarily on study risks and benefits and less on waiving consent or randomization.

EFIC research is of international importance and ethical controversy. This study represents the first attempt to assess views of SCD survivors on this type of research and one of the first to assess patients' views in-depth. Findings indicate broad acceptance of EFIC research among this population and re-focus discussion on what risks are reasonable for non-autonomous subjects. The study also demonstrates potential for valuable input from patients regarding complicated and ethically challenging issues using a method that allows them to develop opinions on unfamiliar issues.

Keywords: bioethics, emergency research, research ethics, informed consent, sudden cardiac death

Introduction

The need for clinical research on treatment of life-threatening and debilitating emergent conditions such as stroke, sudden cardiac death, and major traumatic injury is indisputable. These conditions represent important public health problems, and many current treatments are minimally effective and poorly studied (Cearnal, 2006; Halperin, Paradis, Mosesso, Graham, Sayre, Ornato et al., 2007). Such research, however, poses ethical challenges. Patients are acutely ill and typically unable to make enrollment decisions, and consultation with surrogates is often impossible within an appropriate timeframe.

The U.S. Food and Drug Administration (FDA) and Department of Health and Human Services (DHHS) passed regulations in 1996 permitting an exception from informed consent (EFIC) for certain studies in emergency settings (U.S. Food and Drug Administration, 2004), but few studies have been conducted under these regulations (Hiller, Haukoos, Heard, Tashkin, & Paradis, 2005; Nichol, Huszti, Rokosh, Dumbrell, McGowan, & Becker, 2004). Many investigators have noted difficulty and confusion in implementing the regulations, particularly requirements for community consultation and disclosure (Biros, 2007a; Flynn, 2008; Halperin, Paradis, Mosesso et al., 2007; Kremers, Whisnant, Lowder, & Gregg, 1999; Passamani & Weisfeldt, 2000; Schmidt, 2007). Residual ethical concerns also exist regarding enrolling patients with critical illness in studies without consent, as evidenced by recent controversy surrounding a randomized controlled trial (RCT) of a blood substitute (PolyHeme) in severe hemorrhagic shock (Burton, 2006; Kipnis, King, & Nelson, 2006; Stein, 2004). The FDA (2006) made attempts to clarify regulations and ongoing barriers to EFIC research, but controversy and confusion remain, as evidenced by testimony at a recent public hearing (Academic Emergency Medicine, 2007; Biros, 2007b). Challenges with emergency research are also prominent internationally. In Europe, many countries have developed policies allowing EFIC research after the European Directive on clinical research failed to include provisions to permit it, but such research remains controversial (Lecouturier, Rodgers, Ford, Rapley, Stobbart, Louw et al., 2008).

Despite attention to the ethics of research in emergency settings (Adams & Wegener, 1999; Fost, 1997; McRae, Ackroyd-Stolarz, & Weijer, 2005; Passamani & Weisfeldt, 2000), few data exist regarding how patients view this research (Lecouturier, Rodgers, Ford et al., 2008). Only one study has involved patients enrolled in a trial without consent (Abramson & Safar, 1990); it simply reported that interactions with individuals and families were generally positive when asking for continued participation in a trial of brain resuscitation. The study did not assess participants' views of the research; many did not even understand the study in which they had been enrolled.

Three surveys asked patients in emergency departments their views on EFIC research, all reporting limited support. In two studies, only half of patients were accepting of research involving more than minimal risk (McClure, Delorio, Gunnels, Ochsner, Biros, & Schmidt, 2003; Smithline & Gerstle, 1998) The third study, examining general attitudes rather than particular scenarios, found less than half of participants accepting of EFIC research (Triner, Jacoby, Shelton, Burk, Imarenakhue, Watt et al., 2007). A fourth survey, of Scottish outpatients, found increased acceptance and willingness to be enrolled, particularly if risks are minimal, raising questions about whether EFIC acceptance may vary by geography or culture (Booth, Lind, Read, & Kinsella, 2005). These surveys were all limited by using closed-ended questions about topics that most people have never considered and not exploring participants' reasons for their views or level of understanding of research generally or in emergencies. The latter two in particular contained little to no description of research and how it differs from clinical care. It is difficult to know whether these findings represent informed or considered attitudes.

Two qualitative studies attempted to deepen understanding. Blixen and Agich (2005) conducted in-depth interviews with recent stroke patients. Most supported stroke research and said they would base enrollment decisions primarily on risks and benefits to them. About half had significant concerns with waiving consent, but the nature of those concerns was not examined in depth, nor were particular scenarios. Richardson and colleagues (2005) asked focus group participants to consider several scenarios with varying levels of risk and benefit. They reported general acceptance of low-risk scenarios and wide variation in concerns about EFIC research in general, particularly when involving greater risks. Their data suggest broader acceptance than other studies, and this study is the only one to ask participants to consider scenarios in an open-ended format. Importantly, responses differed from responses to more abstract questions and from survey research. Neither study, however, discussed RCTs.

Finally, some data on public views of EFIC research come from reports of community consultation and disclosure efforts from particular trials and the FDA docket of EFIC research (Mosesso, Brown, Greene, Schmidt, Aufderheide, Sayre et al., 2004; Santora, Cowell, & Trooskin, 1998; Shah & Sugarman, 2003). Most comments received during two-way public disclosure and community consultation have been supportive. Community consultation efforts, however, have rarely been reported, and public disclosure is not often designed to solicit comments.

Because EFIC research may involve significant risks and represents an important departure from common clinical research practice, exploring the nature and range of potential participants' views and understanding is critical. This study deepens knowledge of how a significant and previously unstudied population- survivors of sudden cardiac death (SCD)- view research in emergency settings without informed consent. By using in-depth interviews and asking participants to evaluate particular examples, this study fills an important gap in existing literature and contributes to discussion of appropriate research methodology when conducting empirical research in bioethics.

Methods

This qualitative study enrolled adult survivors of SCD who have an implantable cardioverter-defibrillator (ICD) for prevention of recurrent fatal arrhythmia, an important population for several reasons. First, with the exception of Blixen and Agich (2005), prior studies have not focused on individuals likely to be enrolled in EFIC research. Patients in this study experienced a condition that could have made them eligible for enrollment and may put them at increased risk of having an event making them eligible in the future. Second, personal experience likely helps patients appreciate the importance of treatment for SCD and the serious risks and benefits associated with SCD and its treatment. Finally, their personal experience was expected to facilitate engagement in discussing these issues.

Recruitment

Participants were recruited through an outpatient cardiology clinic at the Johns Hopkins Outpatient Center from early August to December 2005. A Health Insurance Portability and Accountability Act (HIPAA) waiver was obtained to allow identification of eligible patients by searching electronic patient records and clinic schedules. Once identified, clinic staff contacted eligible patients and asked if they would discuss participation with an investigator at their next visit.

The sample was drawn by convenience, and recruitment continued until informational redundancy was reached (when no new themes emerged). Because this population was predominantly older and white, staff were asked to pay particular attention to identifying and calling younger or African-American patients. Recruitment for these patients continued after recruitment of white and older patients had stopped.

Most interviews took place in-person in a private conference room or separate waiting area; two were conducted by telephone. If a patient's spouse was present, the spouse was permitted to participate (spouse responses noted where reported). Patients interviewed in-person provided written informed consent; those interviewed by phone gave verbal consent. This study was considered exempt from review by the Johns Hopkins Bloomberg School of Public Health Committee for Human Research.

Interviews

Interviews were conducted by Neal Dickert using a semi-structured guide containing 4 domains: demographic information; research experience; experience with SCD; and views on emergency research. The guide was developed in consultation with colleagues with expertise in qualitative research and research ethics. Views on research in emergency settings were ascertained using seven scenarios representing a range of risks (minimal to significant), potential for therapeutic benefit (none to significant), and study designs (single-arm, open-label studies to individual and community-level randomized trials). The scenarios included: 1) a single-arm study of a new treatment for SCD; 2) a randomized controlled trial (RCT) comparing a new intervention vs. existing SCD treatment; 3) an RCT comparing two existing SCD treatments; 4) an RCT comparing a blood substitute vs. saline in out-of-hospital treatment of hemorrhagic shock; 5) a site-randomized trial of public access defibrillators; 6) a study presenting no prospect of direct benefit but involving an extra blood draw in the setting of SCD; and 7) a similar study to 6 but using a contrast agent posing a one in 10,000 risk of death. Scenarios were designed by investigators in consultation with colleagues to represent the range of research in emergency settings and to isolate participants' concerns. Two scenarios (numbers 4 and 5) were versions of well-known cases (the Phase III PolyHeme study and the Public Access Defibrillation Trial respectively); others were created by investigators.

Consistent with the semi-structured approach, each scenario was read verbatim to participants. Subsequent questioning varied based on participants' responses. Several participants were not asked to discuss all scenarios. Scenario 3 was added during the study in response to one participant's comments regarding scenario 2. Additionally, scripts for RCT scenarios were altered to explain the process of randomization more clearly after 4 participants did not appear to understand this concept. Denominators reported reflect numbers of participants asked the final form of each question.

Most interviews lasted 30 to 60 minutes. Eighteen interviews were digitally recorded and transcribed for entry into a qualitative data software package (QSR N6) for analysis. One interview was not recorded due to equipment failure. Extensive notes were taken to include these data.

Analysis

This study employed the method of qualitative description (Sandelowski, 2000); the analytic goal was in-depth description of the range and nature of views regarding EFIC research. A two-part coding scheme was developed to analyze data. First, to group data into content areas, a codebook was created based on major study domains and case scenarios. Second, as themes emerged, a set of thematic codes spanning scenarios was developed. After several interviews, colleagues unaffiliated with this project reviewed transcripts and provided comments on themes observed and the adequacy of the emerging coding strategy. New codes were created as themes emerged during interim and final analyses and were grouped into major categories that serve as the basis for this paper's headings. After all data were collected, each transcript was reviewed using the complete thematic codebook. To improve reliability of interpretation, three colleagues reviewed six transcripts and discussed interpretations of participants' comments in major domains. Differences between reviewers' and investigators' interpretations were discussed, and investigators refined analysis where relevant.

Results

Results are provided describing the study population, participants' experience with and attitudes toward research and medicine, and attitudes toward emergency research.

Study Population

Thirty-seven patients were identified. Three were not contacted due to believed inability or unwillingness (according to clinic staff) to be interviewed, and three could not be reached by telephone or letter. Of 31 patients contacted, 22 were willing to participate. Nineteen patients and four spouses were interviewed in total. The study population reflected the eligible clinic population in most demographic categories. Men and women were equally represented. Participants were predominantly well-educated, at least 60 years old, and white, with an average of eight years since SCD (Table 1). About half of patients' primary SCD event occurred in-hospital, typically after myocardial infarction, and half occurred out-of-hospital.

Table 1.

Population demographics

Number (n=19*)
Age Mean 62.1
 <50 2
 51-60 3
 61-65 8
 66-75 5
 >75 1
Gender
 Male 10
 Female 9
Race
 White/Caucasian 16**
 Black/African-American 3**
Education
 Some high school 1
 Graduated high school 4
 Some college 2
 Graduated college 3
 Postgraduate training 7
Time since SCD
 ≤ 5 years 8
 6-10 years 5
 >10 years 6
Religiosity***
 Very 10
 Somewhat 5
 Not at all 1
*

plus 4 spouses

**

plus 2 spouses

***

2 did not answer

Knowledge and Attitudes Regarding Research and Medicine

Familiarity with clinical research

About two-thirds of participants had previously participated in some form of research; most others had never been asked. The range of prior experience varied from completing questionnaires to participating in an RCT. While two participants cited prior experience as a basis for their attitudes toward research, the level of prior experience did not predict the nature of responses to scenarios.

Scenarios were detailed and required careful explanation to ensure understanding and interpretability of responses. The first four participants, in fact, seemed to interpret questions about acceptability of RCTs as asking which treatment arm they would choose. In subsequent interviews, the investigator structured questions more simply and presented randomized scenarios in a step-wise fashion. Understanding substantially improved, allowing greater insight into participants' views.

Positive attitudes

Most participants were positive toward research and felt research in emergency settings is important, in part because they believed they had personally benefited from research.

I think it's okay because science is advancing every day and I mean back 20 years ago, if I would have had my cardiac arrest, I wouldn't be here probably today (317).

This sense of gratitude and acceptance was often accompanied by optimism about the potential for important research advances. Only one participant was pessimistic regarding whether research could change practice, claiming studies are subjective and doctors are

absolute about their particular theory… They're going to do it their way (283).

Altruism

About half of participants expressed altruistic desires. Some stated their desire to help future patients by participating in a particular study. Others stated a more general desire to play a role in improving treatment for patients.

Medical research to a patient is not related in his or her mind to immediate treatment results. Where do I fit into the overall success in my problems… I want to contribute to medical knowledge. (195).

Importantly, altruism was typically a secondary consideration. The primary concern generally was about direct risks and benefits.

Trust in medicine and research community

Most participants exhibited general trust in physicians, with about half stating trust in physicians' ability to determine acceptable risks and benefits. These participants expressed confidence in individual physicians, the medical community, and scientific review.

If Dr. X. or Dr. Y or some of the other folks who are involved in that feel that it's a reasonable step to take… to find out whether it's better for future cardiac patients, I would be okay with that (546).

Two participants and one spouse were less trustful. One recalled stories of terminally ill family members' being “tortured” by investigators. Another, who had been involved in multiple studies and was highly supportive of research, was concerned about trials of new interventions happening “too soon.” She was skeptical of “experimental” treatments, as was one husband, who believed drug company motives' may compromise safety.

Views of Case Scenarios

Seven case scenarios were presented. Most participants were engaged in discussion and supported research in emergency settings, accepting the notion that consent is not always possible (Table 2). In each case, participants were asked whether they thought the scenario was acceptable for researchers to do and for their thoughts about being included personally. Participants were most divided in response to the first case, involving a new intervention to treat SCD. Only half found this scenario acceptable. Participants were generally supportive of other scenarios; however, it was rare for participants to be unhesitatingly positive about scenarios other than the study involving only a blood draw. While they recognized the importance of research in emergency settings and accepted EFIC, participants were concerned about the risks of enrollment and uncertainty of benefit. Similarly, though largely accepting of randomization, many were uneasy about it.

Table 2.

Summary of responses to individual scenarios

Case Description Participants' views Comments
# Accept # Reject # Uncertain # Asked
1) No control arm- New intervention for treatment of SCD 8 7 1 16 Concern about baseline risks and risks and benefits of the new intervention
2) RCT- New Intervention vs. standard treatment for SCD 7 4 2 13 Concern primarily about risks and benefits of experimental intervention.
3) RCT- 2 existing treatments for SCD 7 1 0 12* 1 participant who rejected and 1 who was uncertain about prior scenario explicitly accepted this scenario
4) RCT- New blood substitute vs. saline for severe hemorrhagic shock 13 1 1 19* About half expressed hesitancy or reservation General acceptance of bad outcomes.
5) Public Access Defibrillator (PAD) Study- Site-randomized study of Automated External Defibrillators 10 0 1 13** Respondents' understanding less clear than other scenarios but generally accepting of need for comparison.
6) No prospect of direct benefit-blood draw while resuscitated 18 0 0 18 Most participants very comfortable with this scenario.
7) No prospect of direct benefit- Scan with contrast posing 1/10,000 chance of death 12 3 2 17 Many said that 1/10,000 very slim odds of harm. 2 uncertain best categorized as very reluctant acceptors.
*

As described, the first 4 participants' understanding of this scenario was insufficient for their responses to be interpretable.

**

2 participants' understanding of this scenario was insufficient for responses to be interpretable

Only two participants answered differently regarding the acceptability of a study and their willingness to be enrolled. One was unwilling to be enrolled in an RCT of a new intervention but thought the study should be done, even if enrolling people like her. Another was willing to be treated with an experimental intervention without consent but could not support enrolling others in the study. In contrast, several participants explicitly stated their determination of acceptability was based on personal willingness to be included.

Concern about potential harms and benefits

Consideration of risk and benefit was the central criterion by which participants judged the acceptability of scenarios, and most felt that emergency research should be primarily constrained by patients' interests. Three participants explicitly stated regarding the first scenario that research goals should always be secondary to patients' well-being.

Participants expressed greatest worry about risk in scenarios involving new interventions. About half stated that the likelihood of causing harm was the crucial consideration in determining the acceptability of using a new intervention in at least one scenario.

I don't even like drugs that are approved by the FDA until they've been on the market for a while and you find out what side effects they didn't catch (019).

Significant concern about the risk of new interventions was more often expressed by women than men. The three youngest participants- all of whom were women- all rejected the single-arm scenario as unacceptable on the basis of risk. Similarly, many participants asked specifically about the extent of animal testing and safety studies in scenarios involving new interventions.

If not focused on potential for harm, most participants had concerns about likelihood of benefit from any new intervention; they wanted to ensure any new treatment presented as good or better chances for favorable outcomes as standard treatment.

If it's something that potentially could help me, bring it on. You know and if the upside is better than my downside (127).

The three participants identifying themselves as risk-takers and those generally more positive toward research were more likely to say the likelihood of benefit was the deciding factor. Those more cautious and more likely not to be supportive of a given scenario tended to stress risks. The difference in willingness to accept risk in exchange for benefit is significant, but the two groups, comprising almost everyone, essentially expressed identical concerns- that expected safety and effectiveness of an intervention are the crucial determinants of study acceptability when patients cannot consent.

One important consideration for many participants was the condition-related prognosis and adequacy of existing treatment. Many stated explicitly that background risks are highly relevant in deciding the acceptability of studies of new interventions. Many stated it would be acceptable “if that's the only option to save that life” (261), or if the intervention is a “last ditch effort” (827). Others expressed nuanced views, that their level of concern about experimental treatments depended heavily on the effectiveness of the existing treatment and the prospect of benefit with experimental treatment. As one participant stated, if an existing treatment were to work 5% of the time,

it's kind of like saying that the standard treatment is barely effective and so this new treatment, it's not hard to go over that bar. Where the standard treatment is more, you know, let's say 50%, well then the new treatment really has to be an improvement from there, you know, we have to really have some good background studies that make us think gee this really could help a lot of people (259).

Acceptance of potential for harm

Because it was expected that risks and benefits would be of primary importance, participants were asked their tolerance for potential harms to enrolled patients. They were told that the blood substitute RCT resulted in excess deaths in the intervention group, as happened in the Baxter DCLHb study and likely the recent PolyHeme study (Lewis, Berry, Cryer, Fost, Krome, Washington et al., 2001; Northfield Laboratories, 2006). Participants were asked whether these undesirable outcomes changed their views about the acceptability of the study.

Despite their focus on risk (or deprivation of benefit) in evaluating other scenarios, most responses conveyed an acceptance that risk is unavoidable and an acknowledgment that researchers were not intending to harm anyone.

Well as a layman, your first reaction would be if it was my loved one who died, I would be very, very upset but if you take it the next step and say what if this thing did work and we had a way of saving thousands and thousands of lives throughout the country, maybe millions throughout the world, by having this blood substitute and unfortunately… I would say it seems to me to be a reasonable approach to trying to make saving lives better (546).

One husband stated that he was glad they had given the intervention in the setting of an RCT.

You have to know if it's gonna work and instead of going out there and really giving it to everyone, you know, I think you done it the right way (spouse, 317).

Several raised concerns about legal liability, but only three participants had reactions that reflected non-acceptance of a negative outcome in research.

Acceptance of some risk with no prospect of direct benefit

Because many important studies offer no prospect of direct benefit, participants were asked about two such scenarios with varying levels of risk. Most participants were accepting of the study involving only a blood draw (minimal risk). More surprisingly, 12 found the scenario acceptable involving using a contrast agent, said to have a one in 10,000 risk of death, for purely research purposes. Three participants found this study unacceptable, two of whom consistently opposed imposing risk on patients who cannot consent. The third gave a more qualified rejection, saying such risk would require a particularly important study.

you'd have to convince me first that what we could learn was beneficial enough to really improve things overall for people in the future but, you know, just looking at this part of it and saying okay we want to learn more and 1 in 10,000 people are gonna, you know, not return because of it, that's not worth it to me (259).

Two participants accepting this study similarly insisted that the underlying question should be particularly important. One reluctant acceptor stated it would be disrespectful

to do anything against somebody's knowledge but the overall good, maybe I gotta be a guinea pig (127).

Most participants found a one in 10,000 risk of death small enough to be acceptable. Several then stated that one in 1,000 or one in 100 would be unacceptable. While the investigator reiterated that the study would not benefit the enrollees, it is possible that several participants did not understand that the contrast agent and scan were not clinically indicated.

Acceptance of waived consent

Provided study risks and benefits were considered appropriate, participants accepted that informed consent is often impracticable in emergency settings and did not believe consent is an absolute requirement. Everyone found at least one scenario acceptable, and all involved waiving consent. They did, however, often emphasize that consent should be obtained where possible. Even when finding it acceptable, not all participants felt comfortable with waiving consent.

360: I'm all for learning and people studying and stuff like that, and I guess in some situations, that's the only way you can learn,… by experimenting or doing that even if the people don't have a choice. But I can't say that I'm totally comfortable with it.

And as may be expected given their focus on risks and benefits, participants were more concerned about waving consent where risks were higher.

Acceptance of randomization

The process of randomization is one of the most foreign aspects of research to most laypersons. Randomization is difficult to understand (Robinson, Kerr, Stevens, Lilford, Braunholtz, & Edwards, 2004; Robinson, Kerr, Stevens, Lilford, Braunholtz, Edwards et al., 2005), but most participants seemed to understand what it involves and why researchers might do it, particularly after changes were made to the RCT scripts. The scenario in which patients would be randomized to one of two existing treatments was particularly helpful in discerning views on randomization. This scenario was asked of only eight participants as it was included after a participant suggested his views regarding an RCT of an experimental intervention would change if both interventions were currently used. Only one patient and one spouse objected to this scenario. The patient asserted that doctors ought to make individual treatment decisions, that uncertainty was unacceptable, and that randomizing patients is “disrespectful at a minimum… when life is on the line.” The husband (whose wife accepted randomization) argued that historical controls should be sufficient, stating concern about randomizing ill patients.

You've got to break eggs to make omelets but when you're dealing with the highest form of life on this planet, hey, let's flip a coin, it just seems so, I can't think of a word, it just seems so, thoughtless, you know (spouse, 428).

In contrast, most concerns about RCT scenarios were not about randomization but about risks and benefits of the treatments being compared. For example, 2 participants with objections to an RCT of an experimental vs. standard treatment supported an RCT of 2 existing treatments, with one describing such a study as “essential.”

Interestingly, shifts in attitude toward randomization over the course of interviews were common, possibly reflecting conflicting attitudes or how questions were presented. However, this may also suggest randomization is easier to accept as one becomes more familiar with it (Fallowfield, Jenkins, Brennan, Sawtell, Moynihan, & Souhami, 1998). One participant who initially rejected the idea of an RCT stated, regarding randomization in the blood substitute case:

I think if you're going to do it as part of research and to get the medical profession to agree with it, aren't you going to have to randomize it (195)?

The wife of another participant stated that her attitude evolved over the interview.

I keep saying I don't want to be the guinea pig …but if you don't do the research and you don't get the results and that's the only way to get them, I understand that then it does make sense to do that when you think about it in the long run (spouse, 461).

While shifts in attitude were common and acceptance of RCTs was generally high, much of the acceptance was reluctant. Participants often felt uncomfortable with randomization but acknowledged its scientific importance.

The site-randomized scenario- the public access defibrillator study- also provided insight into views on randomization. Ten of 11 who seemed to understand this study indicated it was reasonable to put automated external defibrillators in some sites and not in others to determine effectiveness. Interestingly, the participant most strongly opposed to individual randomization explicitly accepted this study because it did not involve randomizing someone “in front of you” (624).

Discussion

The clearest message from this study is that participants believed EFIC research can be acceptable in emergency settings. Nobody found such research inherently unacceptable, and the level of acceptance of particular scenarios was higher than previously reported. Thirteen of 15 participants accepted the RCT of a new blood substitute, and 10 of 11 accepted the public access defibrillation study. In contrast, McClure et al. (2003), reported that only 48.5% and 61.5% would be willing to be enrolled in versions of these same studies. Smithline and Gerstle (1998) reported fewer than 50% willing to be enrolled in a study of a novel medication without consent.

Several reasons may explain greater acceptance among this population. These participants were trustful of physicians and grateful for research-proven interventions that positively affected their lives. Additionally, many participants had participated in research before. This factor has been associated with positive attitudes toward research but was not similarly predictive in this study (Comis, Miller, Aldige, Krebs, & Stoval, 2003; Ohmann & Deimling, 2004; Sugarman, Kass, Goodman, Perentesis, Fernandes, & Faden, 1998). This population was also generally more educated, older, and more predominantly white than in studies by McClure, et al. (2003) and Smithline and Gerstle (1998); however, we doubt these differences explain differing acceptance of EFIC research. Smithline and Gerstle, for example, found an inverse correlation between educational status and willingness to participate in EFIC research. McClure et al. found lesser acceptance of EFIC generally among non-whites but no difference in acceptance of scenarios.

A likely more important reason for differing acceptance is this study's design. Because it engaged participants more deeply, asked them to evaluate detailed scenarios, and probed to clarify views, we suspect this study better captured the nature of participants' views. Richardson et al. (2005) similarly reported that responses differed when asked more particular scenarios. These issues are ones most respondents had never considered, and in contrast to surveys, this design allowed individuals to think through scenarios and develop their own views (Mason, 2002). In conducting empirical research soliciting patients' views on complicated ethical issues, it seems crucial to conduct interviews in a way that allows patients to engage deeply enough, and with sufficient understanding, to form and express views in an interpretable manner. This form of questioning need not be limited to exploratory or qualitative research. Larger, quantitatively-oriented projects may also benefit from in-depth interviewing and more open-ended, flexible questioning. This format may introduce greater potential for social acceptability bias (patients' responding in ways they believe will please the investigator), and data are more difficult to manage, but the methodology may yield more interpretable and meaningful findings.

Another important finding is that randomization was not a major source of concern. No previous study has examined views of randomization in this setting. Randomization is difficult to explain, and simply knowing that assignment is done “at random” does not indicate real understanding (Edwards, Lilford, & J, 1998; Robinson, Kerr, Stevens et al., 2004; Robinson, Kerr, Stevens et al., 2005). However, many participants did seem to understand reasons for randomization, and the increase in understanding and acceptance over the course of interviews suggests the concept may have become more familiar, a phenomenon noted in at least one other study with cancer patients (Fallowfield, Jenkins, Brennan et al., 1998). This finding is encouraging, as it suggests potential participants can be involved in a meaningful discussion about the acceptability of RCTs with proper explanation. It also suggests that community consultation- required for EFIC research in the US- may yield meaningful input if done properly.

While not as concerned about randomization, participants were concerned with the relative risks and benefits posed by different study arms. It is thus not surprising participants generally accepted the RCT comparing two existing treatments. This acceptance has important implications. The regulatory mechanism for approving these studies in the U.S. is unclear due to requirements that existing treatments be unsatisfactory or unproven and that studies present a prospect of direct benefit (Passamani & Weisfeldt, 2000; U.S. Food and Drug Administration, 2004; Watters, Sayre, & Silbergleit, 2005). It may be possible to categorize these trials as posing no more than minimal risk (thus being allowed by the general section of the Common Rule), but it is not clear many IRBs would make that interpretation (U.S. Department of Health and Human Services,; Watters, Sayre, & Silbergleit, 2005). If not, many low risk studies may not be approvable under regulations intended to promote clinical trials and protect subjects. Given the scientific need for comparsion studies, their relative safety, and their apparent acceptability to patients, it is important that their permissibility be clarified. The general acceptance of this type of lower-risk study may also provide some support for the proposal by the American Heart Association that community consultation requirements be scaled according to the level of risk studies pose (Flynn, 2008; Halperin, Paradis, Mosesso et al., 2007).

It was surprising that most participants found acceptable the scenario involving a contrast agent with a one in 10,000 risk of death for purely research purposes. Although scenarios differed, Richardson et al. (2005) reported similar willingness to consider research posing more than minimal risk and presenting no prospect of direct benefit. Whether participants truly accept this risk is unclear; risk assessment is complicated (Tversky & Kahneman, 1987). However, a study conducted without consent and presenting no prospect of direct benefit must involve no more than minimal risk according to U.S. regulations, a standard that has been endorsed in emergency research (McRae & Weijer, 2002; U.S. Department of Health and Human Services; U.S. Food and Drug Administration, 2004). Although interpreting the minimal risk standard is contentious (Karlawish, 1996), few IRBs are likely to consider a one in 10,000 risk of death minimal. Imposition of such risks without a prospect of benefit has been dismissed by some writers as unethical (McRae & Weijer, 2002), but apparent acceptance by patients prompts close examination of this important issue.

The over-arching finding in this study may be least surprising but raises a significant challenge. Participants' emphasis on the risk-benefit ratio as essentially determinative of acceptability begs for more careful consideration of what levels of risk are reasonable, particularly when imposed for societal benefit on participants who cannot consent. Many stated that individual interests should not be compromised, but most acknowledged that research involves risks that might not otherwise exist, reflecting a tension at the heart of research ethics. Determining what constitutes reasonable risk in emergency settings is a project with serious policy implications (Dickert & Sugarman, 2007). Estimating risk and benefit are technical tasks best done by the medical community, but determining the reasonableness of risk-benefit tradeoffs is a value judgment that should consider the views of people likely to be enrolled (Brody, 1995; Karlawish, 1996). The criterion of clinical equipoise, suggested in the regulations and prominently invoked in research ethics literature, unfortunately offers thin guidance. It is difficult to operationalize, may over- or under-protect, and may not prevent exploitation, particularly if reduced to mere statistical uncertainty (Miller & Brody, 2003).

Community consultation may help to incorporate prevalent views into assessments, but little is known about how to use community consultation to achieve this goal, and deep questions exist about how to incorporate “public opinion” into ethical assessments. Studies like this help deepen understanding of what potential enrollees consider to be reasonable and suggest in-depth conversations will be needed to engage people effectively and understand the nature of their concerns.

Translating these and future findings into policy decisions about how much risk and benefit are acceptable will require significant conceptual work. It is insufficient to make decisions about acceptability of EFIC research on the basis of public opinion. Yet it is equally important not to disregard the considered judgments of persons potentially affected by such research.

Limitations

This study population was small, and participants were generally older, white, and well-educated. In addition, the population was recruited from one U.S. urban academic cardiology clinic and may differ from the populations of community-based clinics or clinics in other countries. It is certainly possible, that views on this type of research differ in countries with a less individualistic focus on health care (Booth, Lind, Read et al., 2005). No important differences in response were obvious based on participants' age, education, or race, but the goal of the study was not to identify such associations. Furthermore, the relationship of education, age, and race to attitudes toward research is complicated and inconsistent (Comis, Miller, Aldige et al., 2003; Corbie-Smith, Thomas, & St. George, 2002; Corbie-Smith, Thomas, Williams, & Moody-Ayers, 1999; Larson & McGuire, 1990; Madsen, Holm, & Riis, 1999; Shavers, Lunch, & Brumeister, 2002; Sugarman, Kass, Goodman et al., 1998; Trauth, Musa, Siminoff, Jewell, & Ricci, 2000). Transferability of these data to very different populations may thus be limited, and future work should seek to engage these populations. Nevertheless, the population likely reflected survivors of SCD in the United States, given the unfortunate social and racial disparities in survival of SCD (Iwashyna, Christakis, & Becker, 1999).

Conclusion

This exploratory, qualitative study deepens understanding of how survivors of SCD view research in emergency settings without informed consent. It revealed greater support than previously demonstrated and illustrated that most concerns center around reasonable risks and benefits. Given the need to conduct RCTs comparing existing treatments, participants' responses suggest a need for regulatory clarity regarding these studies. The suggestion that increases over minimal risk without prospect of direct benefit may be acceptable to many patients at least prompts further examination of what levels of risks are ethical in this setting. Perhaps the clearest implication, however, is the need for ongoing dialogue among potential participants, investigators, and institutional review boards about the reasonability of particular risks and expectations of benefit in this important, but challenging research. This study suggests that such discussions may be possible and helpful when conducted in a way that is sensitive to the complexity of these issues.

Acknowledgments

We would like to acknowledge Holly Taylor in particular for her assistance in developing the instrument and methodology used in this study, as well as Hilary Bok, Ruth Faden, Katherine Smith, and Jeremy Sugarman for their valuable comments on the manuscript. We also thank Misti Capps and Kathy DeVaughn for their assistance in identifying and recruiting participants.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Contributor Information

Dr. Neal Dickert, Emory University Atlanta, GA UNITED STATES, neal.dickert@emory.edu

Nancy E Kass, Johns Hopkins University Bloomberg School of Public Health and Phoebe R. Berman Bioethics Institute, ndicker@jhmi.edu.

References

  1. Academic Emergency Medicine. 2007;14(4):e27–e73. doi: 10.1197/j.aem.2006.12.002. [DOI] [PubMed] [Google Scholar]
  2. Abramson NS, Safar P. Deferred consent: use in clinical research. Annals of Emergency Medicine. 1990;19(7):781–784. doi: 10.1016/s0196-0644(05)81703-2. [DOI] [PubMed] [Google Scholar]
  3. Adams J, Wegener J. Acting without asking: an ethical analysis of the Food and Drug Administration waiver of informed consent for emergency research. Annals of Emergency Medicine. 1999;33:218–233. doi: 10.1016/s0196-0644(99)70398-7. [DOI] [PubMed] [Google Scholar]
  4. Biros MH. Food and Drug Administration public hearing on the conduct of emergency clinical research: testimony of the Society for Academic Emergency Medicine. Acad Emerg Med. 2007a;14(4):e27–e29. doi: 10.1197/j.aem.2006.12.002. [DOI] [PubMed] [Google Scholar]
  5. Biros MH. Struggling with the rule: the exception from informed consent in resuscitation research. Acad Emerg Med. 2007b;14(4):344–345. doi: 10.1197/j.aem.2006.12.007. [DOI] [PubMed] [Google Scholar]
  6. Blixen C, Agich G. Stroke patients' preferences and values about emergency research. Journal of Medical Ethics. 2005;31:608–611. doi: 10.1136/jme.2004.010488. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Booth MG, Lind A, Read E, Kinsella J. Public perception of emergency research: a questionnaire. European Journal of Anesthesiology. 2005;22:933–937. doi: 10.1017/S0265021505001596. [DOI] [PubMed] [Google Scholar]
  8. Brody BA. Ethical issues in drug testing, approval, and pricing: the clot-dissolving drugs. New York: Oxford University Press; 1995. [Google Scholar]
  9. Burton TM. Red flags: amid alarm bells, a blood substitute keeps pumping. The Wall Street Journal. 2006:A1. [PubMed] [Google Scholar]
  10. Cearnal L. The birth of the NETT: NIH-funded network will launch emergency neurological trials. Annals of Emergency Medicine. 2006;48(6):726–728. doi: 10.1016/j.annemergmed.2006.10.010. [DOI] [PubMed] [Google Scholar]
  11. Comis RL, Miller JD, Aldige CR, Krebs L, Stoval E. Public attitudes toward participation in cancer clinical trials. Journal of Clinical Oncology. 2003;21:830–835. doi: 10.1200/JCO.2003.02.105. [DOI] [PubMed] [Google Scholar]
  12. Corbie-Smith G, Thomas S, Williams M, Moody-Ayers S. Attitudes and beliefs of African Americans toward participation in medical research. Journal of General Internal Medicine. 1999;14:537–546. doi: 10.1046/j.1525-1497.1999.07048.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Corbie-Smith G, Thomas S, St George D. Distrust, race, and research. Archives of Internal Medicine. 2002;162:2458–2463. doi: 10.1001/archinte.162.21.2458. [DOI] [PubMed] [Google Scholar]
  14. Dickert NW, Sugarman J. Getting the Ethics Right Regarding Research in the Emergency Setting: Lessons from PolyHeme. Kennedy Institute of Ethics Journal. 2007 doi: 10.1353/ken.2007.0010. forthcoming. [DOI] [PubMed] [Google Scholar]
  15. Edwards SJ, Lilford R, H J. The ethics of randomised controlled trials from the perspective of patients, the public, and healthcare professionals. British Medical Journal. 1998;317:1209–1212. doi: 10.1136/bmj.317.7167.1209. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Fallowfield L, Jenkins V, Brennan C, Sawtell M, Moynihan C, Souhami R. Attitudes of patients to randomised clinical trials of cancer therapy. European Journal of Cancer. 1998;34(10):1554–1559. doi: 10.1016/s0959-8049(98)00193-2. [DOI] [PubMed] [Google Scholar]
  17. Flynn G. Community consultation for emergency exception to informed consent: how much is enough? Annals of Emergency Medicine. 2008;51(4):416–419. doi: 10.1016/j.annemergmed.2008.02.007. [DOI] [PubMed] [Google Scholar]
  18. Fost N. Waived consent for emergency research. American Journal of Law and Medicine. 1997;24(23):163–181. [PubMed] [Google Scholar]
  19. Halperin H, Paradis NA, Mosesso VNJ, Graham N, Sayre MR, Ornato J, Gerardi M, Nadkarni VM, Berg R, Becker L, Siegler M, Collins M, Cairns CB, Biros M, Vanden Hoek T, Peberdy MA. Recommendations for implementation of community consultation and public disclosure under the Food and Drug Administration's ‘exception from informed consent requirements for emergency research.’. Circulation. 2007;116:1855–1863. doi: 10.1161/CIRCULATIONAHA.107.186661. [DOI] [PubMed] [Google Scholar]
  20. Hiller KM, Haukoos JS, Heard K, Tashkin JS, Paradis NA. Impact of the Final Rule on the rate of clinical cardiac arrest research in the United States. Acad Emerg Med. 2005;12(11):1091–1098. doi: 10.1197/j.aem.2005.04.025. [DOI] [PubMed] [Google Scholar]
  21. Iwashyna T, Christakis N, Becker L. Neighborhoods matter: a population-based study of provision of cardiopulmonary resuscitation. Annals of Emergency Medicine. 1999;34(4):459–468. doi: 10.1016/s0196-0644(99)80047-x. [DOI] [PubMed] [Google Scholar]
  22. Karlawish JHT. The controversy over emergency research: A review of the issues and suggestions for a resolution. American Journal of Respiratory and Critical Care Medicine. 1996;153:499–506. doi: 10.1164/ajrccm.153.2.8564087. [DOI] [PubMed] [Google Scholar]
  23. Kipnis K, King NMP, Nelson RM. An open letter to IRBs considering Northfield Laboratories' PolyHeme trial. American Journal of Bioethics. 2006 doi: 10.1080/15265160600685580. [DOI] [PubMed] [Google Scholar]
  24. Kremers MS, Whisnant DR, Lowder LS, Gregg L. Initial experience using the Food and Drug Administration guidelines for emergency research without consent. Annals of Emergency Medicine. 1999;33(2):221–229. doi: 10.1016/s0196-0644(99)70399-9. [DOI] [PubMed] [Google Scholar]
  25. Larson E, McGuire D. Patient experiences with research in a tertiary care setting. Nursing Research. 1990;39(3):168–171. [PubMed] [Google Scholar]
  26. Lecouturier J, Rodgers H, Ford GA, Rapley T, Stobbart L, Louw SJ, Murtagh MJ. Clinical research without consent in adults in the emergency setting: a review of patient and public views. BMC Medical Ethics. 2008;9(9) doi: 10.1186/1472-6939-9-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Lewis RJ, Berry DA, Cryer HI, Fost N, Krome R, Washington GR, Houghton J, Blue JW, Bechofer R, Cook T, Fisher M. Monitoring a clinical trial conducted under the food and drug administration regulations allowing a waiver of prospective informed consent: The diaspirin cross-linked hemoglobin traumatic hemorrhagic shock efficacy trial. Annals of Emergency Medicine. 2001;38:397–404. doi: 10.1067/mem.2001.118223. [DOI] [PubMed] [Google Scholar]
  28. Madsen S, Holm S, Riis P. Ethical aspects of clinical trials: the attitudes of the public and out-patients. Journal of Internal Medicine. 1999;245(6):571–579. doi: 10.1046/j.1365-2796.1999.00502.x. [DOI] [PubMed] [Google Scholar]
  29. Mason J. Qualitative Researching. Thousand Oaks: SAGE Publications; 2002. [Google Scholar]
  30. McClure KB, Delorio NM, Gunnels MD, Ochsner MJ, Biros MH, Schmidt TA. Attitudes of emergency department patients and visitors regarding emergency exception from informed consent in resuscitation research, community consultation, and public notification. Academic Emergency Medicine. 2003;10:352–259. doi: 10.1111/j.1553-2712.2003.tb01348.x. [DOI] [PubMed] [Google Scholar]
  31. McRae AD, Weijer C. Lessons from everyday lives: A moral justification for acute care research. Critical Care Medicine. 2002;30(5):1146–1151. doi: 10.1097/00003246-200205000-00032. [DOI] [PubMed] [Google Scholar]
  32. McRae AD, Ackroyd-Stolarz S, Weijer C. Risk in emergency research using a waiver of/exception from consent: implications of a structured approach for institutional review board review. Acad Emerg Med. 2005;12(11):1104–1112. doi: 10.1197/j.aem.2005.04.005. [DOI] [PubMed] [Google Scholar]
  33. Miller FG, Brody H. A critique of clinical equipoise: therapeutic misconception in the ethics of clinical trials. Hastings Center Report. 2003;33(3):19–28. [PubMed] [Google Scholar]
  34. Mosesso VNJ, Brown LH, Greene HL, Schmidt TA, Aufderheide TP, Sayre MR, Stephens SW, Travers A, Craven RA, Weisfeldt ML PAD Trial Investigators. Conducting research using the emergency exception from informed consent: the Public Access Defibrillation (PAD) Trial experience. Resuscitation. 2004;6(1):29–36. doi: 10.1016/j.resuscitation.2003.11.016. [DOI] [PubMed] [Google Scholar]
  35. Nichol G, Huszti E, Rokosh J, Dumbrell A, McGowan J, Becker L. Impact of informed consent requirements on cardiac arrest research in the United States: exception from consent or from research? Resuscitation. 2004;62(1):3–23. doi: 10.1016/j.resuscitation.2004.02.013. [DOI] [PubMed] [Google Scholar]
  36. Northfield Laboratories. Northfield Laboratories reports preliminary top-Line data in pivotal Phase III trauma study 2006 [Google Scholar]
  37. Ohmann C, Deimling A. Attitude towards clinical trials: results of a survey of persons interested in research. Inflammation Research. 2004;53(Supplement 2):S142–S147. doi: 10.1007/s00011-004-0353-6. [DOI] [PubMed] [Google Scholar]
  38. Passamani ER, Weisfeldt ML. Task Force 3: special aspects of research conduct in the emergency setting: waiver of informed consent. Journal of the American College of Cardiology. 2000;35:862–880. [PubMed] [Google Scholar]
  39. Richardson LD, Wilets I, Ragin DF, Holohan J, Smirnoff M, Rhodes R, Winkel G, Rodriguez M, Ricci E. Research without consent: community perspectives from the Community VOICES study. Acad Emerg Med. 2005;12(11):1082–1090. doi: 10.1197/j.aem.2005.06.008. [DOI] [PubMed] [Google Scholar]
  40. Robinson EJ, Kerr C, Stevens A, Lilford R, Braunholtz D, Edwards S. Lay conceptions of the ethical and scientific justifications for random allocation in clinical trials. Social Science and Medicine. 2004;59:811–824. doi: 10.1016/s0277-9536(03)00255-7. [DOI] [PubMed] [Google Scholar]
  41. Robinson EJ, Kerr CE, Stevens AJ, Lilford RJ, Braunholtz DA, Edwards SJ, Beck S, Rowley M. Lay public's understanding of equipoise and randomisation in randomised clinical trials. Health Technology Assessment. 2005;9(8) doi: 10.3310/hta9080. [DOI] [PubMed] [Google Scholar]
  42. Sandelowski M. Whatever happened to qualitative description? Research in Nursing & Health. 2000;23(24):334–340. doi: 10.1002/1098-240x(200008)23:4<334::aid-nur9>3.0.co;2-g. [DOI] [PubMed] [Google Scholar]
  43. Santora TA, Cowell V, Trooskin SZ. Working through the public disclosure process mandated by use of 21 CFR 50.24 (Exception to Informed Consent): guidelines for success. The Journal of Trauma: Injury, Infection, and Critical Care. 1998;45:907–913. doi: 10.1097/00005373-199811000-00012. [DOI] [PubMed] [Google Scholar]
  44. Schmidt TA. Food and Drug Administration public hearing on the conduct of emergency clinical research: testimony of Dr. Schmidt. Acad Emerg Med. 2007;14(4):e59–e62. doi: 10.1197/j.aem.2006.11.028. [DOI] [PubMed] [Google Scholar]
  45. Shah AN, Sugarman J. Protecting research subjects under the waiver of informed consent for emergency research: Experiences with efforts to inform the community. Annals of Emergency Medicine. 2003;41:72–78. doi: 10.1067/mem.2003.14. [DOI] [PubMed] [Google Scholar]
  46. Shavers VL, Lunch CF, Brumeister LF. Racial differences in factors that influence the willingness to participate in medical research studies. Annals of Epidemiology. 2002;12:248–256. doi: 10.1016/s1047-2797(01)00265-4. [DOI] [PubMed] [Google Scholar]
  47. Smithline HA, Gerstle ML. Waiver of informed consent: a survey of emergency medicine patients. Academic Emergency Medicine. 1998;16(1):90–91. doi: 10.1016/s0735-6757(98)90074-2. [DOI] [PubMed] [Google Scholar]
  48. Stein R. The Washington Post. Washington, DC: 2004. An experiment in saving lives; emergency patients unwittingly get artificial blood; p. A01. [PubMed] [Google Scholar]
  49. Sugarman J, Kass N, Goodman S, Perentesis P, Fernandes P, Faden R. What patients say about medical research. IRB. 1998;20(4):1–7. [PubMed] [Google Scholar]
  50. Trauth J, Musa D, Siminoff L, Jewell J, Ricci E. Public attitudes regarding willingness to participate in medical research studies. Journal of Health and Social Policy. 2000;12(2):23–43. doi: 10.1300/J045v12n02_02. [DOI] [PubMed] [Google Scholar]
  51. Triner W, Jacoby L, Shelton W, Burk M, Imarenakhue S, Watt J, Larkin G, McGee G. Exception from informed consent enrollment in emergency medical research: attitudes and awareness. Acad Emerg Med. 2007;14(2):187–191. doi: 10.1197/j.aem.2006.08.019. [DOI] [PubMed] [Google Scholar]
  52. Tversky A, Kahneman D. The framing of decisions and the psychology of choice. Science. 1987;211(4481):453–458. doi: 10.1126/science.7455683. [DOI] [PubMed] [Google Scholar]
  53. U.S. Department of Health and Human Services. Title 45 (Code of Federal Regulations), Part 46. Protection of Human Subjects, Subpart D. [PubMed] [Google Scholar]
  54. U.S. Department of Health and Human Services. Title 45 (Code of Federal Regulations), Part 46. Protection of Human Subjects. [PubMed] [Google Scholar]
  55. U.S. Food and Drug Administration. Title 21 (Code of Federal Regulations), Part 50.24 Protection of Human Subjects 2004 [Google Scholar]
  56. U.S. Food and Drug Administration. Guidance for Institutional Review Boards, clinical investigators, and sponsors: Exception from informed consent requirements for emergency research. Rockville, MD: U.S. Food and Drug Administration; 2006. [Google Scholar]
  57. Watters D, Sayre MR, Silbergleit R. Research conditions that qualify for emergency exception from informed consent. Acad Emerg Med. 2005;12(11):1040–1044. doi: 10.1197/j.aem.2005.06.022. [DOI] [PubMed] [Google Scholar]

RESOURCES