The evaluation of the risks and benefits of phase II cancer clinical trials by institutional review board (IRB) members: a case study

Abstract

Objectives

There are indications that institutional review board (IRB) members do not find it easy to assess the risks and benefits in medical experiments, although this is their principal duty. This study examined how IRB members assessed the risk/benefit ratio (RBR) of a specific phase II breast cancer clinical trial.

Participants and methods

The trial was evaluated by means of a questionnaire administered to 43 members of IRBs at six academic hospitals and specialised cancer centres in the Netherlands. The questionnaire addressed: identification and estimation of inconvenience, toxicity, psychosocial distress, and benefits of trial participation to patients; identification and estimation of benefits to future patients and medical science; assessment of the trial's RBR; and assessment of its ethical acceptability.

Results

Most IRB members expected trial participation to involve fairly or very serious inconvenience, fairly severe to sometimes life‐threatening toxicity, and serious psychological and social consequences. Conversely, the perceived likelihood of benefits to patients was modest. Most regarded the study as important, and the balance between risks and benefits to be favourable, and believed that the protocol should be approved. The IRB members' final judgement on the trial's ethical acceptability was significantly correlated with their RBR assessment of the protocol.

Conclusions

Because most patients who participate in clinical trials hope this will prolong their lives, it is suggested that patient information should better describe the anticipated benefits—for example, the likelihood of prolonging life. This would allow patients to make decisions regarding participation based on realistic expectations.

There are indications that institutional review board (IRB) members do not find it easy to assess the ratio of risks to benefits in medical experiments, although this is one of their principal duties.¹,²,³,⁴,⁵ IRBs are legally required to evaluate whether the risk/benefit ratio (RBR) is “reasonable”, “proportional”, or “favourable”. Since there is no consensus on the content of, and categories and criteria for, evaluating the RBR, IRB evaluations, although usually based on long term clinical experience, are unavoidably subjective and intuitive. The lack of shared categories and criteria makes it difficult to trace and discuss differences of opinion within an IRB, and thereby reduces the chances that evaluation of the ratio between risks and benefits will play a prominent role in the final decision regarding the ethical acceptability of the research. The difficulties with assessing the RBR sometimes induce IRBs to leave the evaluation primarily to the potential research participants, arguing that it is their right to determine whether, from their perspective, the relation of risks to benefits is reasonable. Whether this is ethically acceptable, and whether very ill patients are capable of making such stressful decisions, remains unclear.⁶,⁷,⁸,⁹,¹⁰,¹¹

The most important contributions to the study of RBR assessment of medical research involving humans have been those of Levine and Meslin.¹,²,³,¹²,¹³,¹⁴,¹⁵,¹⁶,¹⁷ Levine developed a set of categories to distinguish between risks and benefits for participating patients, for future patients, and for society at large (scientific progress).¹⁴ These categories have been further refined by distinguishing between different kinds or dimensions of risks and benefits for participating patients—for example, physical, psychological, social. Meslin devoted special attention to risk assessment by IRB members.¹ He identified three “decision‐making processes” in risk assessment: identification, estimation, and evaluation of risks. Using Levine's and Meslin's distinction between different types of risks and benefits, between “risk” and “harm”, certain and uncertain risks, and between the phases of identification, estimation, and evaluation of risks, we designed a survey to study IRB members' assessment of the RBR for a specific phase II cancer clinical trial.

The current study represents the second stage of a four‐stage project examining the assessment of the RBR of phase II and III cancer clinical trials by IRB members in the Netherlands. The first stage consisted of semistructured interviews with 53 IRB members regarding the RBR assessment of phase II and III cancer clinical studies in general.¹⁸,¹⁹ This second stage of the research addresses two primary questions: (1) what risks and benefits do IRB members identify in a phase II breast cancer trial, and how do they estimate and evaluate these risks and benefits? and (2) what is their assessment of the RBR and the ethical acceptability of the protocol?

Participants and methods

Study participants, protocol evaluation, and questionnaire

The study was conducted in the period 1998–1999. The IRBs of eight Dutch academic hospitals and specialised cancer centres were asked to participate. We did not select non‐academic hospitals because they do not evaluate sufficient numbers of cancer clinical trials to be appropriate candidates for such a study. Five of the eight IRBs agreed to participate. All members of these IRBs (n  =  64) were invited to take part in the first stage of the study, of whom 52 agreed to do so. One member of the IRB of a sixth Dutch teaching hospital also agreed to participate. The primary reason for non‐participation was constraints on time. Of these 53 IRB members, 10 declined to continue participation in this second stage of the research, again, due largely to time constraints. The final study sample on which the current analysis was based included 43 members of six IRBs in six Dutch hospitals (the Utrecht Academic Hospital, the Vrije Universiteit Medical Center in Amsterdam, the Rotterdam Academic Hospital, the Leiden Academic Hospital, the Netherlands Cancer Institute/Antoni van Leeuwenhoek Hospital in Amsterdam, and the Daniel den Hoed Hospital in Rotterdam). The sample consisted of oncologists (21%), other medical specialists (19%), family physicians (5%), nurses (19%), and other disciplines (36%), including four pharmacists, two ethicists, two behavioural scientists, one statistician, and others. The respondents' age ranged from 28 to 69 years. The majority were men (65%). Nine percent had been IRB members for less than 1 year, 47% for 1–4 years, 23% for 4–7 years, and 21% for longer than 7 years.

An American phase II breast cancer clinical trial—which had been approved in 1997—was selected for the study from the website of the National Cancer Institute. (The trial was used with permission of the authors and is entitled: A pilot study of allogeneic peripheral blood stem cell transplantation for patients with metastatic or recurrent breast cancer using a conditioning regimen of busulfan and cyclophosphamide.) We decided against a European or a Dutch protocol to avoid the chance of distortions in the results due to the possible familiarity of the research participants with the protocol. This phase II trial was designed to investigate the efficacy and toxicity of allogeneic stem cell transplantation in combination with high doses of chemotherapy for patients with metastatic breast cancer. We selected a trial involving a very intensive treatment because we believed this to be the most effective approach to investigating the process of risk/benefit assessment. This trial differs from the usual phase II studies because it was expected to have high risks as well as potentially large benefits, while most phase II trials are expected to be less risky as well as less beneficial. The rationale for using this “atypical” trial was as follows. We expected that such a trial would result in more variation in IRB members' assessments of the RBR and of the ethical acceptability of the protocol than a study with high risks but small benefits or vice versa, or a study with low risks as well as small benefits. The protocol evaluation was conducted by means of a questionnaire consisting primarily of closed‐ended questions. We used closed questions because open questions about the protocol evaluation will be asked in in‐depth interviews in the third phase of the research project. The questionnaire had been previously pilot tested among five IRB members or former IRB members, all of whom participated in the main study. The respondents were asked to study the protocol and the patient information, to complete the questionnaire and to return it by mail. The procedures took approximately 2 hours: 1 hour for studying the protocol and 30–45 minutes to fill in the questionnaire.

The following topics were included in the questionnaire: (1) identification and estimation of the inconvenience, toxicity, psychosocial distress, and benefits of trial participation to patients; (2) identification and estimation of benefits to future patients and medical science; (3) assessment of the overall RBR of the study; and (4) assessment of the ethical acceptability of the study. The questions asked were based on the literature and can be found in the Appendix.¹,²,³,¹²,¹³,¹⁴,¹⁵,¹⁶,¹⁷ Although this protocol does not provide any data about psychological and social risks, we asked whether IRB members believed these types of risk were present in the study. The questions concerning the identification and estimation of toxicity were based on a predetermined list of toxicities drawn from the protocol itself. The specific aspects of possible treatment toxicity that were assessed included: likelihood, severity, duration, reversibility, and amenability to treatment. The only aspects considered to be relevant for assessing the psychosocial burden of the treatment were likelihood, severity, and duration. The only aspects considered to be relevant for assessing the benefits of treatment were likelihood, duration, and importance. The importance of the benefits of the treatment was assessed directly, while that of the risks was captured by the severity rating.

It could be argued that the details presented on the rating of the toxicity and benefits would be more relevant if a comparison were made with the actual “facts” as stated in the protocol. However, in this study such a comparison was not of primary interest or the focus of the research.

Statistical analysis

Descriptive statistics were generated with the SPSS software program. The χ² statistic was used to test the relationship between the IRB members' assessment of the RBR of the protocol and their assessment of the protocol's ethical acceptability.

Results

Identification and estimation of risks and benefits

Inconvenience

As shown in table 1, hospital admission and time investment (travel, waiting, etc.) were rated as the most inconvenient aspects of trial participation.

Table 1 Estimation by IRB members of the inconvenience of the phase II study for participating patients (n  =  43) (%).

	Very	Fairly	Not very	Not at all	Not applicable*
Hospital admission	35	58	7	0	0
IV treatment	5	28	65	2	0
Additional examinations	17	56	27	0	0
Extra control visits	7	49	39	0	5
Time investment (travel, waiting, etc.)	23	49	23	5	0

*Respondents could choose this alternative when they believed a certain form of inconvenience was not present in the trial.

Toxicity

Table 2 presents the evaluation of the most common toxicities along five axes: likelihood, severity, duration, reversibility, and amenability to treatment. There was broad agreement among IRB members on the expected toxicity of the treatment: hair loss, diarrhoea, nausea, vomiting, fatigue, and organ toxicity were estimated by most IRB members as very to fairly likely. Additional toxicities expected by IRB members to be very or fairly likely were: mucositis, infection, fertility problems/damage to offspring, bleeding, change of skin colour, high blood pressure, tremors, painful hands and feet, stomatitis, haematuria, genetic disturbances, and rejection of donor bone marrow.

Table 2 Estimation by IRB members of the likelihood, severity, duration, reversibility, and amenability to treatment of possible toxicity to patients during and/or after the experimental treatment (n  =  43) (%).

	Likelihood			Severity					Duration			Reversibility			Amenability to treatment
	Very high + fairly high	Low + very low	Don't know	Mild	Fairly severe	Severe	Life threatening	Don't know	Some days to some weeks	Some months to years	Don't know	Reversible	Sometimes irreversible + irreversible	Don't know	Amenable to treatment	Not amenable to treatment	Don't know
Hair loss	84	7	9	33	20	37	0	10	17	73	10	79	13	8	9	88	3
Diarrhoea	75	13	12	3	63	22	2	10	86	2	12	76	17	7	93	5	2
Nausea	81	10	9	7	45	38	0	10	87	5	8	80	15	5	98	0	2
Vomiting	86	3	11	2	45	42	0	11	89	3	8	84	8	8	90	5	5
Fatigue	79	14	7	6	48	38	0	8	15	75	10	22	68	10	4	88	8
Organ toxicity (heart, kidney, liver)	63	28	9	0	22	35	33	10	32	30	38	12	73	15	27	48	25
Cognitive neurological problems	37	58	5	15	28	38	3	16	31	44	25	22	50	28	11	52	37
Other	35	58	7	5	37	21	26	11	56	22	22	52	37	11	44	39	17

Most respondents rated these toxic effects as fairly severe to life threatening, expected hair loss and fatigue to last for some months to years, and 30% and 44%, respectively, expected organ toxicity and cognitive neurological problems to last for some months to years. Although hair loss, diarrhoea, and nausea and vomiting were expected to be reversible, fatigue and organ toxicity were typically not. In addition, half of the respondents expected cognitive/neurological problems to be irreversible or sometimes irreversible. Although most expected some toxic effects to be amenable to treatment, for hair loss and fatigue this was not the case. Furthermore, half of the IRB members expected organ toxicity and cognitive/neurological problems, and more than one‐third other toxicity, not to be treatable.

Psychosocial distress

Ninety‐three percent of the IRB members believed trial participation would entail psychological distress for patients beyond that caused by the illness itself (data not presented in tabular form). As shown in table 3, approximately 50% expected patients to experience depression, 79% stress, and 88% uncertainty as a result of trial participation. Other forms of psychosocial distress, such as loneliness, donor dependence, and fear were identified by 16% of the IRB members.

Table 3 Estimation by IRB members of the likelihood, severity, and duration of possible extra psychosocial burden to patients during and/or after the experimental treatment (n  =  43) (%).

	Likelihood					Severity					Duration
	Very high + fairly high	Low + very low	Don't know	Not applicable*	Did not answer question	Not severe	Fairly severe + severe + very severe	Don't know	Not applicable*	Did not answer question	Some days to some weeks	Some months to years	Don't know	Not applicable*	Did not answer question
Psychological burden
Depression	51	35	5	9	0	13	65	11	11	0	22	43	24	11	0
Anxiety	79	12	0	9	0	8	80	2	10	0	35	40	15	10	0
Uncertainty	88	3	0	9	0	7	81	2	10	0	14	64	12	10	0
Other	16	70	5	9	0	0	50	17	33	0	8	42	17	33	0
Social burden
Extra strain on relationship with partner	61	9	2	28	0	3	62	5	30	0	8	45	17	30	0
Extra burden on other social contacts	63	7	2	28	0	7	58	7	28	0	5	49	18	28	0
Loss of prestige	5	60	7	28	0	29	10	22	39	0	7	13	40	40	0
Other	7	63	2	28	0	0	25	0	75	0	0	25	0	75	0

*Percentage of IRB members that did not expect an extra psychosocial burden on patients or did not know this; they did not answer the questions about the likelihood, severity, and duration of extra psychosocial burden.

Seventy‐two percent of the respondents believed trial participation to be a social burden for patients (not presented in tabular form). As reported in table 3, two‐thirds expected a strain on relationships with partners and on other social contacts. Seven percent also mentioned long periods of illness or a strain on patients' professional life as expected stressors.

Benefits to participating patients

As indicated in table 4, 45% of respondents expected tumour remission and 35% a longer symptom‐free period to be fairly or very likely. Only a minority of IRB members expected that other benefits would accrue to patients (except for hope).

Table 4 Estimation by IRB members of the likelihood, duration, and importance of the possible benefits of the experimental treatment to the participating patients (n  =  43) (%).

	Likelihood				Duration				Importance
	Very high + fairly high	Low + very low + zero	Don't know	Did not answer question	Some days to some weeks	Some months to years	Don't know	Did not answer question	Very great + fairly great	Very little + fairly little	Don't know	Did not answer question
Tumour remission	45	43	12	0	7	63	30	0	62	30	8	0
Prolongation of life	16	75	9	0	11	62	27	0	69	26	5	0
Longer symptom‐free period	35	58	7	0	17	59	24	0	71	24	5	0
Less toxicity than alternative treatment	10	76	14	0	24	16	60	0	62	19	19	0
Less pain	7	72	21	0	16	28	56	0	65	19	16	0
Better quality of life	7	72	21	0	7	43	50	0	65	19	16	0
Hope	65	28	7	0	11	64	25	0	58	31	11	0
Other	7	93	0	0	0	67	33	0	67	0	33	0

Benefits to future patients and science

Sixty‐eight percent of the IRB members were unable to estimate how many patients in the Netherlands would benefit annually from the experimental treatment, should it prove effective. Most rated the clinical trial as fairly to very important (8% of very great importance, 39% of great importance, 39% of moderate importance).

Final assessment of the risk/benefit ratio and the ethical acceptability of the research

Thirty percent of the IRB members believed that the risks of the protocol outweighed the benefits, 21% believed that the benefits outweighed the risks, and 35% assigned approximately equivalent weights to the risks and benefits. Thirty‐seven percent said they would approve the protocol and 44% would recommend approval after revision. Although 44% of the IRB members believed that the risks outweighed the benefits or were unable to evaluate the RBR, only 18% said they would reject the protocol or could not judge its ethical acceptability. There was a significant relationship between their assessment of the RBR and the ethical acceptability of the trial (p < 0.031). Most of the IRB members (83%) who believed that the risks of the protocol outweighed the benefits said they would reject the protocol; 17% of them would approve the protocol or would approve it after revision. More than half (54%) of those who believed the benefits outweighed the risks, or who assigned approximately equivalent weights to the risks and benefits, said they would approve it; less than half would reject it or would approve it after revision.

Discussion

The aims of the current study were to examine how individual IRB members assess the diverse risks and benefits of a specific phase II cancer protocol, and to examine how they come to their final assessment of the RBR and the ethical acceptability of the proposed trial. First, we were interested in determining what type of risks and benefits IRB members identify in evaluating a particular phase II cancer protocol, and how they estimate and evaluate these risks and benefits. The results indicate that most IRB members felt competent to estimate specific aspects of the risks and benefits such as likelihood and severity (although the expected duration of such risks and benefits proved more difficult to evaluate), to determine the RBR, and to assess the ethical acceptability of the trial. These findings are consistent with those reported previously for IRB members' estimations for phase II cancer protocols in general.¹⁸,¹⁹

The results also indicate that, besides inconvenience and fairly severe to sometimes life‐threatening physical risks (toxicity), these IRB members identified several serious psychological and social risks of trial participation. This is in line with the distinction made by Levine between different types of risks for patients participating in medical experiments (physical, psychological, social).¹⁴ The results further indicate that, while IRB members believed the research to be important, they expected only modest benefits to accrue to the participating patients. Although a substantial percentage rated benefits to patients such as tumour remission (45%), a longer symptom‐free period (35%), and hope (65%) to be fairly or very likely, only a few expected this to be the case with respect to prolongation of life, reduction in pain, less toxicity than an alternative treatment (for example, no treatment), and a better quality of life.

Secondly, we investigated the IRB members' assessment of the RBR and the ethical acceptability of the protocol. Most believed that the benefits outweighed the risks, or that risks and benefits had nearly equal weight, and most wanted the trial to take place. Furthermore, as one would expect, a significant association was observed between assessment of the RBR and assessment of the ethical acceptability of the trial. This means that the IRB members' evaluation of the RBR played a significant role in their final decision regarding the ethical acceptability of the trial.

Most cancer patients participate in trials because they hope for a treatment effect in terms of prolongation of life.⁶,²⁰,²¹ Because we did not study the letter of informed consent, we cannot say what patients were told about their potential benefit in the trial. However, considering that patients themselves have to carry the risks and the weight they assign to prolongation of their life, it can be doubted whether patients having the same information as the IRB members would come to a RBR assessment similar to that of the IRB members. This observation underlines the need to be open and honest when informing patients, and it provides a very strong argument for taking the perspectives of patients into account when determining the RBR of trials.

A number of the study's limitations should be mentioned. First, we evaluated only one specific phase II cancer protocol. Although it would have been preferable to ask the IRB members to evaluate a range of protocols, this was not feasible given the rather labour intensive nature of the research. We believe that the trial protocol we selected for review was reasonably representative of phase II cancer clinical trials in general, although, as we stated earlier, the treatment schedule was quite intensive.

Secondly, the participants were all drawn from the IRBs of Dutch academic hospitals or specialised cancer centres. Whether our results can be generalised to other types of hospitals or to other countries is uncertain. The settings in which European and American IRBs operate may differ in certain respects. For example, in the USA there is generally a greater concern with protecting hospitals and physicians from possible legal action than is the case in Europe. However, the structure, objectives, and procedures of IRBs are similar, regardless of whether they are American or European. Thus we are fairly confident that our results can reasonably be extended to IRBs in general.

Finally, we would note that our study was based on questionnaire rather than observational data, and that we queried individual members rather than investigating the IRBs as a whole. We realise that the decisions taken by IRBs are often collective ones, and emerge from discussions and debates that take place during IRB meetings. Nevertheless, each IRB member brings his or her own perspective to such deliberations, and is expected to be well prepared to participate actively in the decision‐making process. At the same time, we would recognise the value of other types of research—for example, observational studies, that would better be able to capture the group dynamics involved in IRB decision making. In a future article we will report the results of the latter stage of the current study in which such observational techniques were employed.

Summary

First, most IRB members estimated hope to be as important as physical benefits to patients (about 65% believed physical benefits to be fairly or very important, see table 4), but more likely (65% believed hope to be fairly or very likely versus 7–45% for the physical benefits, see table 4). Hope is, of course, important. Although we did not study why patients want to participate in clinical trials, the literature shows that most do so hoping that this will prolong their lives.²¹ This hope is in danger of becoming irrational when the chances for survival are very low. We observe that the IRB members considered survival benefit to be unlikely compared with tumour remission and a longer symptom‐free period, but they regarded the psychological benefit of hope to be as important as physical benefits in their RBR assessment. If hope is an almost illusionary benefit, is it ethically acceptable to include it in assessing the RBR? We know from earlier studies that patients' and medical doctors' expectations of benefit from participation in medical experiments are different.²⁰ It would be an improvement if patient information sheets paid more attention, as they normally do to the risks, to the possible benefits to participants. This would prevent patients' hope from becoming irrational.

Secondly, although the IRB members evaluated the risks to be fairly severe to sometimes life‐threatening, most believed that the benefits outweighed the risks, or that risks and benefits had a nearly equal weight, and most wanted the trial to take place. Their final judgement on the protocol's ethical acceptability was significantly correlated with their RBR assessment of the protocol. Thus, evaluation of the risks and benefits actually formed part of the IRB members' final evaluation of the protocol's ethical acceptability.

Abbreviations

IRB - institutional review board

RBR - risk/benefit ratio

IDENTIFICATION AND ESTIMATION OF INCONVENIENCE FOR THE RESEARCH PARTICIPANTS

“How burdensome do you think this situation is for the patient?”

Admission of the patient to the hospital, IVs, extra examinations, extra control visits, and time investment (travel, waiting, etc.).

(Answers were scored on a 4‐point Likert scale from “very burdensome” to “not burdensome at all”.)

IDENTIFICATION AND ESTIMATION OF TOXICITY FOR THE RESEARCH PARTICIPANTS

“What kind of toxicity do you believe would happen during and/or after the research?”

They could choose between hair loss, diarrhoea, nausea, vomiting, fatigue, organ toxicity, cognitive neurological problems, or other toxicity.

(If a certain kind of toxicity was believed not to happen, this answer was scored as an answer to the next question concerning the likelihood of the toxicity as that the likelihood would be “not high at all”. If the toxicity was believed to happen, the respondent was asked how he or she estimated the likelihood of this toxicity and his or her answer was also scored as an answer to the next question concerning the likelihood of the toxicity.)

“Indicate the likelihood, severity, duration, reversibility and amenability to treatment for the patient during and/or after experimental treatment of the following: hair loss, diarrhoea, nausea, vomiting, fatigue, organ toxicity, cognitive neurological problems, or other toxicity.”

(Answers were scored on a 4‐point Likert scale from: (1) “very high” to “not high at all”; (2) “mild” to “life threatening”; (3) “a few days” to “years”; (4) “reversible” to “irreversible”; and (5) “amenable to treatment” to “not amenable to treatment”.)

IDENTIFICATION AND ESTIMATION OF PSYCHOSOCIAL DISTRESS FOR THE RESEARCH PARTICIPANTS

“Do you believe participation in the experimental treatment will impose an extra psychological burden (beyond the burden imposed by the illness and previous treatment) for the patient?”

(Answers were scored as: (1) “yes”; (2) “no”; (3) “don't know”.)

“Do you believe participation in the experimental treatment will place an extra social burden (beyond the social burden imposed by the illness and previous treatment alone) on the patient?”

(Answers were scored as: (1) “yes”; (2) “no”; (3) “don't know”.)

“Indicate the likelihood, severity, and duration of the following psychosocial stress for the patient during and/or after the experimental treatment as an extra psychological and social burden (over and above the stress related to the illness and treatment): depression, stress, uncertainty, extra strain on relationships with partners, extra strain on other social contacts, and loss of prestige.”

(Answers were scored on a 4‐point Likert scale: (1) “very high” to “not high at all”; (2) “not so severe” to “very severe”; (3) “a few days” to “years”.)

IDENTIFICATION AND ESTIMATION OF BENEFITS TO THE RESEARCH PARTICIPANTS

“What kind of benefits do you think participating patients will obtain during and/or after the research?”

They could choose between tumour remission, prolongation of life, longer symptom‐free periods, less pain, better quality of life, hope, or other benefits.

(If a certain kind of benefit was believed not to happen, this answer was scored as an answer to the next question concerning the likelihood of the benefit as that the likelihood would be “not high at all”. If the benefit was believed to happen, the respondent was asked how he or she estimated the likelihood of this benefit and his or her answer was also scored as an answer to the next question concerning the likelihood of the benefit.)

“Indicate the likelihood, duration, and importance of the following benefits for the patient during and/or after experimental treatment: tumour remission, prolongation of life, longer symptom‐free periods, less toxicity than alternative treatment, less pain, better quality of life, hope, or other benefits.”

(Answers were scored on a 4‐point Likert scale: (1) “very high” to “not high at all”; (2) “a few days” to “years”; (3) “very important” to “not important at all”.)

IDENTIFICATION AND ESTIMATION OF BENEFITS TO FUTURE PATIENTS AND MEDICAL SCIENCE

“Indicate how many patients would benefit every year in the Netherlands from the experimental treatment should it prove effective.”

“Indicate the importance of the research study.”

(The first was an open question; answers to the second question were scored on a 4‐point Likert scale from “very important” to “not important at all”.)

RISK/BENEFIT RATIO ASSESSMENT OF THE STUDY

“What is your final assessment of the risk/benefit ratio of the study?”

(Possible answers included: (1) “the risks outweigh the benefits”; (2) “the benefits outweigh the risks”; (3) “the risks and benefits weigh approximately the same”; or (4) “don't know”.)

ASSESSMENT OF THE ETHICAL ACCEPTABILITY OF THE STUDY

“What is your final assessment of the study?”

(Possible answers included: (1) “approve”, (2) “needs revision”; (3)”reject”; (4) “don't know”.)