Trials and tribulations: the professional development of surgical trialists

Abstract

BACKGROUND

Regulatory and professional bodies issue an ever-increasing number of guidance documents on the ethics and methods of clinical trials, but the quality of clinical trials of invasive therapeutic procedures continues to be a concern. We interviewed aspiring and accomplished surgical trialists to understand how they use guidance documents and other resources in their work.

METHODS

We performed a qualitative research study involving semistructured interviews of a diverse sample of 15 surgical trialists.

RESULTS

Professional development as a surgical trialist was haphazard, inefficient, and marked by avoidable mistakes. Four types of resources played constructive roles: formal education; written materials on clinical trials; experience with actual trials; and interpersonal interactions with peers, experts, collaborators, and mentors. Recommendations for improvement centered on education, mentoring, networking, participating in trials, and facilitation by department chairs.

CONCLUSIONS

The haphazard and unstructured nature of the current system is adding unnecessarily to the numerous challenges faced by surgical trialists.

The gold standard research design for the generation of evidence about the efficacy of therapeutic interventions is the randomized controlled trial, but the small number of such trials published every year on invasive therapeutic procedures indicates this method of evaluation is still far from the norm for such procedures. Although the dearth of surgical trials is an issue that has been recognized for decades¹ and criticized as a barrier to improving clinical care,² the proportion of surgical publications reporting results of randomized trials has increased little over the past 20 years.³ In a recent review of nearly 38,000 reports published between 1999 and 2008 that reported evaluations of the effects of invasive therapies for 33 different conditions, we found that only 8% could be classified as comparative clinical trials. The remainder used weaker study designs. Moreover, although the quantity of surgical trials is problematic, we found, as others have,^4,5 that the quality of some of the surgical trials that are performed, even those that are the most highly cited, is undermined by methodologic deficiencies and/or poor reporting.⁶

Flaws in trials in surgical and other clinical trials cannot be attributed to a lack of how-to documents. Even if text-books are disregarded, many international and national regulatory agencies such as the Food and Drug Administration (FDA), professional societies, research sponsors, advisory groups, and other highly regarded entities issue guidance documents promulgating ethical and methodologic standards for clinical trials of therapeutic modalities. In a systematic search for guidance documents produced between 1949, the year the Nuremberg Code was issued, and 2008, we identified 1,004 regulatory or normative documents providing guidance on clinical trials—302 from the United States alone—and 39 of the total 1,004 that addressed topics specific to trials of invasive procedures and/or medical devices. In fact, there has been an almost exponential growth in the issue of these regulatory and normative documents over time. Of the 1,004 documents our search identified, 20 were created or updated between 1949 and 1989, 146 between 1990 and 1999, 194 between 2000 and 2002, 253 between 2003 and 2005, and 316 between 2006 and 2008 (date of issue was not provided for 75 documents). In sum, more than three quarters of the documents were released or updated in the past decade. All are available on the Internet.^7,8

The situation in which the field finds itself seems paradoxical. On the one hand, guidance documents on how best to design and conduct clinical trials are being issued at an ever-increasing rate by influential agencies and societies. On the other, the design and conduct of many surgical trials continue to be stubbornly substandard. This led us to question, to what extent do surgical trialists use these guidance documents? If not these documents, what resources are being used in the design and conduct of surgical trials? Accordingly, we conducted a qualitative study of aspiring and accomplished surgical trialists, with the goal of understanding how such individuals use various information and education sources in their work in clinical trials. Our respondents answered the questions we posed, but also additionally provided information on a broader and much more important issue: the process by which surgical trialists are educated about how to perform clinical trials of surgical procedures. Our findings have important implications for leaders in surgery, for individuals who aspire to become surgical trialists, and for those responsible for them.

Methods

This qualitative research study was approved by the Institutional Review Board of The Methodist Hospital Research Institute. The sponsor had no role in study design, data collection/analysis, or writing the article.

A list of potential interviewees was developed based on the senior author’s knowledge of surgical trialists and supplemented with the names of trialists identified in an earlier phase of the parent research project.⁷ Potential participants purposefully were selected based on credentials in academic medicine and activity in the field of surgical trials, and additionally aiming for diversity in experience and specialty. Potential participants were invited by e-mail to be interviewed for the study, and interested individuals were confirmed via e-mail and scheduled for an interview. We did not attempt follow-up contact with nonresponders. Participation was voluntary. Participants provided informed consent to be interviewed, and they did not receive any compensation for their time.

Semistructured interviews were conducted by 2 authors (C.M.A. and A.F.J.) from December 2010 to March 2011. Interviews were conducted in face-to-face meetings (n = 10) or over the telephone (n = 5) and each lasted no more than 1 hour. Interviews were audiorecorded and transcribed verbatim, and transcripts were reviewed for accuracy. Identifiers were removed, and specific information in the transcript that could be used to identify the respondents (eg, institutional affiliation or involvement in a particular study) was redacted or changed to nonspecific content.

The interview was designed to elicit information regarding respondents’ use of clinical research resources (such as the guidance documents described earlier and any others) and their perspectives on the practical utility and educational value of various resources and experiences, including how they were sought or applied in a specific clinical trial. The interview protocol was divided into 3 parts: clinical or professional training, clinical research training and education, and experience and use of various kinds of research resources (eg, printed materials) during involvement in a particular surgical trial. We also asked respondents to give recommendations pertinent to future surgical trialists.

All transcripts were read independently by 2 investigators (C.M.A. and A.F.J.) to gain an overall impression of the content. An initial set of content codes identifying themes or units of meaning were derived from the first 5 transcripts. Coding took place as interviews were completed; accrual of interviewees was stopped at 15, when no new themes emerged from the latest interview transcript. Codes were refined iteratively over the course of review of the entire set of 15. After each transcript was reviewed and coded by each reviewer, the 2 reviewers met to discuss individual transcripts and fragments. Coding disagreement was evaluated and reconciled. Next, the coded fragments relevant to each theme were extracted from individual transcripts and compiled into separate data sets for further analysis of themes, specification of themes, and patterns. The two original reviewers then were joined by a third reviewer (N.P.W.) for analysis of the compiled fragments. Within each thematic category patterns and apparent conflicts among fragments were discussed, and the important elements were identified and summarized. Finally, relationships between codes were identified.

Results

Characteristics of the sample

The final sample of 15 interviewees included 13 physicians, 11 of whom are surgeons, a research ethicist, and a biostatistician. Four interviewees are women. All are US-based. Specialties represented are general surgery, cardiothoracic surgery, internal medicine, neurosurgery, pediatrics, urogynecology, urology, and vascular surgery. Years since attainment of the doctor of medicine or doctor of philosophy degree ranged from 13 to 64; roughly a fourth of the sample were fewer than 20 years out, 20 to 30 years out, 30 to 40 years out, and 40 or more years out. Of the 11 surgeons, 9 currently hold a current leadership role such as a deanship, chairmanship of an academic department, or clinical service chief position. Nine of the 11 surgeons had devoted 1 or more years to research during their surgical residency or fellowship, but these years were devoted almost exclusively to laboratory rather than clinical research. In addition, some of the physician respondents had voluntarily sought out formal training relating to clinical research, including formal study that led to an advanced degree (eg, Master of Science) (n = 5) or graduate school coursework not leading to a degree but relevant to clinical trials (n = 1). Four surgeons had participated in brief seminars (lasting days or weeks) in clinical trial methods offered by organizations such as the American College of Surgeons or the Veterans Affairs Cooperative Studies Program. All 15 respondents had completed the periodic mandatory human subjects research credentialing course required by their respective universities. Eleven of the respondents were accomplished trialists (had played a key role in ≤1 completed and highly cited surgical trials), and 4 were attempting to launch their first significant trial.

Themes that emerged from interviews

Four broad learning themes emerged from the transcripts: (1) reflections on personal development as a trialist; (2) learning from written materials (hard copy or virtual hard copy online materials); (3) experiential learning, which had several subcodes, namely learning by doing investigator-initiated, industry-initiated, and service-based learning, defined as learning accrued via the experience of serving on an Institutional Review Board (IRB), a Data Safety Monitoring Board, a grant application review committee, or reviewing manuscripts for journals; and (4) learning from people, defined as learning from direct interpersonal interaction with peers, experts, collaborators, mentors, as well as from anonymous reviewers who provided the learner with comments on an IRB application, grant application, or application to the Food and Drug Administration. In addition, respondents provided recommendations on how the structure and process of surgical clinical trial education and performance might be improved.

Reflections on personal development as a trialist

In recounting their personal development as a trialist, nearly all the respondents used words denoting difficulty (eg, “painful,” “learning by making mistakes,” “through error”) (Table 1). The other common subtheme was the haphazard or at least informal nature of educational and developmental experiences (eg, “self-trained,” “self-education,” “catch-as-catch-can,” “had to do things on my own”). In addition, although none of the respondents who had a formal education in clinical trials thought it was sufficient, those who did not expressed regret or wistfulness about the lack of formal education. One respondent told us, “I had no formal training. I learned it by absorption. Basically by starting with case series, case controls, databases, interacting with experts and mentors. That’s how I assimilated my knowledge and my background in research.” In describing how to overcome obstacles in a clinical research career, this same individual said, “My advice to at least my partner … keep knocking your head against the wall until the wall breaks down. You’ve got to keep trying … It’s easy to say. ’hey, it’s hard,’ and not to do it. But if you keep persisting there is always a way, I think. There’s always a way.”

Table 1.

Respondents’ reflections on their personal development as a surgical trialist

“… It’s very painful to start to do these trials on your own and then sort of learn as you go, as you’re doing it. That becomes  a very painful experience.”

“So the training largely has been one of’catch as catch can,’ [what] one gets as a [specialty] fellow and as a junior clinician;  secondarily, when one joins an academic institution there’s a human subjects research credentialing that one has to  take … I’ve completed those in all those institutions where I’ve been a part of, where I’ve trained. In addition, as a junior  member of [name of society] [I took] their introduction to clinical trials for …junior investigators [on] the safe and ethical  conduct of clinical trial research in human subjects. So those are the primary things that I’ve done. But otherwise it [training  in clinical trials] has been largely ’catch as catch can’ and really piecing together the ethics, professionalism, the safety  standards that one gets at these institutions, coupled with just interacting with people as you go along.”

“There wasn’t any formal research education really, beyond a course in biostatistics in medical school I guess. But I had a lot of  informal mentoring from people whom I respected a lot.”

“Then I guess much of what I was able to learn was self education, just sort of learning by doing and making mistakes and  trying to fix them.”

“So while there wasn’t you know, formal didactic instruction at that time on how to conduct and manage clinical trials, there  was the ability to read the protocols, see the patients, and follow the progress of the clinical trials and see how they were  done. And Dr. [name] was also a very prolific writer of grants. And so I spent a lot of time, you know, helping prepare grants and developing data for the grants  and reviewing patients who had been in clinical trials for various publications.”

Learning from written materials

Most respondents did not cite written how-to materials, whether online or in hard copy, as a principal source of information (Table 2). Efforts to answer questions about trial design or conduct or to find information by consulting written materials (contextual, clinical, ethical, methodologic, or regulatory) frequently were described as frustrating. When questioned about whether they referred to any clinical trial guidance documents, all but one respondent stated that such materials had not been accessed or used during the development and conduct of a surgical trial or in the individual attempts to become educated about clinical research. Only 3 respondents spontaneously named a guidance document (the Common Rule,⁹ the International Council on Harmonisation series,¹⁰ and Consolidated Standards of Reporting Trials [CONSORT¹¹] Statement).

Table 2.

Respondents’ views on learning from written materials

Lack of availability, awareness, or usefulness of written documents “… there’s not much [in the literature] in terms of surgical  interventions and the risk, at least not much that you could find. They’re probably out there but they’re not easily findable or  searchable.”

“I don’t think any of them [the printed research ethics material] discussed the issue. I  mean this was really out of left field, I guess, except for [one other trial], and theirs was slightly different…”

“I don’t know where the methodology for this is in any document we did. I don’t think we [had a] good textbook for  randomized trials and design. I’m not aware that we did.”

Benefits or lack thereof from reading published literature as groundwork for a clinical trial

“We looked at the literature … at what the current accepted best [perioperative] practices were. There were a lot of places  where there were best practices. There were some places where there were no  best practices. So places where there were no known best practice, we let [participating centers] basically do what they were currently doing without trying to change their  practice, because it would have been a whole other level to try to not just do the study intervention but also change the  practice at any one hospital. We felt that that would negatively impact patient enrollment at the hospitals, and since there  was no other guidance as to what the best practice was we didn’t feel that there was any reason for trying to change the  practice at any one place.”

“… there had been a few studies done before of [procedure]. Actually I wrote a little critique of those studies way  back … that expressed my frustration with the lack of scientific rigor of the studies that had been done and summarized  what I saw as the problems with those studies, mainly that they were done on [patients] who … if I remember correctly,  didn’t really come near meeting, for the most part, any reasonable minimum standards for surgery. They were just sort of  collected. They compared [patients] who had surgery with [patients] who didn’t have surgery without too much regard for  what the indications had been.”

“My study had never been done before. So certainly there wasn’t anything there [in the literature as examples or guidance]. I  did not look up surgical intervention trials. Take that back. I did, after I came back here and the first time I got slammed. I  did look it up again. And then you get a bunch of stuff that was not helpful. I did go to clinical trials.gov to see the active  clinical trials but you know that’s not that helpful either for their design aspect… it would be nice to have had a review  article or a resource where it was written about how to design a surgical clinical trial. Like that was the title [of the published  article].”

“In order to answer that [clinical] question, I started looking through the literature and I saw that there were very small case  reports about this but no large case series for me to refer to see if this was reasonable for my patients.”

“There’s a lot of documentation out there, sure. I mean I looked at previous research protocols to go through and see how they  were set up. And obviously the whole published literature on clinical trials in [condition]. Then we looked at you know  protocols that were in progress, like we went to the [cooperative group name] site and downloaded their protocols … And  it’s all published on the web so you can download all of that. You can look at it. There are obviously NIH resources that you  can go to look for how you deal with all of these regulatory issues.”

Views on required human subjects research credentialing process

“It [the human subjects research credentialing training from institution] didn’t help me for this [trial], did not. I mean those  sorts of things are important and mandatory and compulsory. And I recognize why. Maybe they should have [been helpful  during the trial]. Maybe I’m not smart enough to figure out how they should have. I view those things as more ethics  involved.”

“I think the [institution] has gone to Collaborative Institutional Research Training (CITI) training, and I think that training is  very reasonable, and it hits all of the high points. Certainly it’s not in depth, and I think the classroom training from the  Fundamentals [of Clinical Trials Course] is an important supplement to that. But I think to get the basics across. I think the  CITI training is pretty good.”

“I’ve had to go through that [CITI training] I guess [several times] now because I’ve been here for [N] years on faculty … But  that has kind of minimal educational value I would say. It’s more of a testing thing. It’s not a ’here’s how you do it’ deal.”

“I learned as I went through the CITI course. It was not testing knowledge that I already had. I basically had to read through  the course and figure out what was right and wrong and then took the test and passed and said that I was certified. And you  know there were questions in there that I thought were somewhat helpful in designing a study. You know, they’d ask you the  difference between a phase one, phase two, phase three trials and basic things like that. But they certainly didn’t go through  like say a grant writing class would break it down into sections and say, ’This is how you write a grant.’”

Reasons given by respondents about why they had not used written resources fell into 4 categories. First was the belief that what they needed did not exist. “There could’ve been a resource that examined surgical interventions in clinical trials … [about] a new surgical intervention in a current condition, or old surgical intervention in a new clinical condition or maybe some combinations thereof … and how one approaches each one. There wasn’t anything out there like that.” Second was the suspicion that written information sources on surgical trials might exist but currently was unknown to the individual. “I suspect … there’s probably lots of stuff that exists that I’m just not aware of … it might be that there is a gazillion documents, there’s a huge book of things waiting for me in a closet that I just don’t know about.” Third was that the available, accessed documents failed to address the specific research issues at hand. “In the process of designing this trial, I did a huge literature review and then worked with [collaborators] … [then it] was basically me literally sitting there arguing back and forth with [collaborators], for liter-ally a year, about every single line in there. Every single line was a whole argument and discussion … just going back and forth and back and forth. There was really no one to guide us [on the research topic of interest] because there was no modern literature … Everything was very sparse.” Fourth was the sentiment that written literature was not a desirable medium for education on certain topics. “In terms of … how do you work with [subjects], how do you get them to answer the quality of life and at the same time answer questions about their sex life as I’m looking at their toes, and you know that stuff … I don’t think it’s in the books or if so I’m not sure I want to read it.”

Some interviewees who had conducted searches for instructional guidance believed that the clinical literature was a source of information because it allowed them to critique previous flaws and mistakes in other trials. “… reading the literature … you saw how it was done wrongly or not so good … I tried to learn from those mistakes … [as] somebody that was a voracious reader of the literature you know, quietly critical in my own mind of the way trials were being done … I mean there’s just … profound amounts of bias in the design and implementation of surgical trials.” Related to this was a perception that written protocols from previous trials had practical utility as how-to guidance on setting up a new trial.

Experiential learning

All respondents reported experiential learning as a highly valuable or even critical aspect of their research education, and, for some, a type of instruction that could not have been achieved by any other means (Table 3). Learning by performing trials and service-based learning were the 2 subtypes of experiential learning.

Table 3.

Respondents’ views on experiential learning

Learning by doing or designing surgical trials

“[Trying to conduct research in a clinical center] it’s kind of seen as you know, are you here to kind of scoop up these patients?  So I had to be very careful to say I’m not here for taking patients away and that kind of stuff. And that was a huge challenge  to kind of get through.”

“In the process of kind of thinking through that [clinical research] project… all kinds of things came up that I really hadn’t  thought about. Like the consent process. Oh my Lord, I mean, to me the consent process should be pretty straightforward  because I’m not doing anything invasive to these people. But it turned out, of course, to be a multi-page document and we  were going to the [centers] to make it convenient for the patients. Well then they wanted their own consent form plus our  consent form etc. And then other things that I hadn’t really considered … I didn’t think about it but should we be  considering educational level in these patients you know? Are we considering you know kind of how do they feel about  themselves and doing quality of life indices and those things?”

“There are a lot of things, like … the trial initiation compliance with regulatory requirements… approval of the research  protocol by the Department of Defense, Human Research Protection Office, and harmonization of IRB requirements of each  center with the human research protection office, appointment of a central trial monitor, appointment of local trial monitors,  trial initiation meeting of all investigators, writing and signing of site agreements between the sponsor and the individual  hospitals, and so on. So you know there’s a lot of steps to go through. And if you haven’t done it I mean … you don’t know  what you’re getting yourself into.”

“[A colleague] actually suggested initially why don’t you try this for an RO1 type grant? And I said ok, well maybe, I never really  thought about that. Why not? So then … what about trial design? How are we going to design it you know? This is a surgery that  has been done before … So number one, who is going to pay for it? Are insurance companies going to pay for it because being  experimental, is it an experimental technique? No, but it’s a technique that’s experimental in this setting. So we called Medicare and  they said no way we’re going to pay for this. And another insurance company said no. And that is, it’s much more expensive than  you can imagine than most pills because you have the preoperative evaluation that takes place. The [operating room] OR time that  takes place. The postoperative care, hospitalization that might take place and then potential complications that one has to  anticipate, that patients might come back again. So figuring out a per patient cost for a surgical intervention and various things  that one might do to prepare for the surgical intervention was going to be [a] tidy sum of money. So to figure out how are we  going to do this and then the power of the trial might still be informative and etc … nonetheless I didn’t know any better and I  just … went to a particular individual, the head of the division, and we worked it out. ”

“So I think we all have, surgeons all have ideas of how we might be able to help someone. But I think a systematic resource of  how one can implement those, that’s statistically robust and ethically sound, ain’t available …#########. what I did originally was in  retrospect, really naive. I had a reasonable idea. But not really well thought out in terms of precisely of all the bad things  that could happen and then quantitating [sic] those and then yes, it really wasn’t. So now I look back on the second version  [of the grant application] I think my goodness. This is like night and day.”

“I thought I was being sensitive to the ethical nature of this trial. But perhaps clearly to some I was not enough. So I think  had the design been different initially it would have conveyed to the reviewers that I had a better appreciation for the ill  effects for the possible intervention.”

“… bottom line is what we learned from that trial is that powering is critical and number two that surgical trials are so  expensive that it’s very difficult to power a trial adequately unless you have a dramatic [change] from base line.”

“And I’ve actually written a couple of protocols that are unusable. And I wrote one protocol that got $70,000 of funding and  has yet to enroll a patient, [N] years later, which I think is a sin … our protocol is broken and we’re in the process of  revising it and trying to rewrite it. But had I known what I know now I would have written it completely different. And it  would have been better use of funding. So I think it actually was a mistake that we got funded. We didn’t really know what  we were doing … If you are a surgeon, you want to make sure that every surgery you do is the best you can make. Having  junior people who are inexperienced write protocols and do research can probably as much harm as do good.”

“I became the local investigator for the trial in [place]. And through that trial I just learned so much about clinical trials because you  know we had our conference calls every week, bigger conference calls every month. I became aware of what a data safety &  monitoring board is about. I became much more familiar with all of the IRB issues, change in protocol, protocol deviations. And  then I also attended all of the investigators meetings that took place over the course of five years. You know that ended up being  one of the best clinical trials that was ever conducted in general surgery. The main outcome was published in the [name of journal]  and we got about ten papers out of the trial … So it was a very good learning experience for me to just as a naive person going  and requesting to be a site investigator to the point where you know I became a more seasoned investigator throughout the trial.”

Learning by doing industry-initiated trials

“The [clinical condition] testing that we used for our screening study I basically took directly from one of our industry  sponsored trials, their methodology and that kind of stuff. The sad part about that is as we started doing it, and I went back  to them and I said, ’Ok, so how are you exactly scoring this now that you’re collecting all of this data?’ which perhaps I  should have asked beforehand. They had no idea how to score it. I mean they didn’t know if out of the nine points was one  going to be positive or did you mean two out of nine to be positive. Like I had no idea. I was like oh God, what was I  thinking. So I think there’s a little too much leap before we look carefully, a mentality probably particularly in surgeons.”

“I mean, they [industry trials] generate some funds which then you can turn and use for your other stuff. But you also, if you  become pretty active in those, get to go to the investigator meetings and get involved in the design of the next study, which  that’s been I think really good for a couple of these things that I’m interested in … you know, what should the end points  be, what is the FDA accepting or not accepting. And that’s helpful.”

“The two [people] that were the lead authors of [journal names] papers were there at those [industry sponsored trial] meetings,  and talked about the hardships in the study design and the agony over what was the primary endpoint going to be and the  fights back and forth with the FDA. And that was, I’m sure, painful for them to relive but it was really educational for me, as  a relatively young investigator trying to design my own trials and think about you know, where I’m going to go.”

“… I would say that in my area there are not enough investigators doing investigator-initiated trials really. Everybody’s doing  industry-sponsored trials where you’re handed the protocol, you search for the patients and go from there. And one of my  goals if I’m to stay in academics is really to generate some investigator-initiated trials. It’s sort of a trial and error process in  doing so though, right?”

“I feel like through error I’ve got a pretty good grasp on a lot of these things now, as I get spanked on monitoring visits, etc.,  etc … I go to enough investigator meetings to get this.”

“… some of the people involved are strictly there with industry and to make a buck and some of the people are there to really  make a great trial.”

Service-based learning

“I was invited to be on the [name] study section of the NIH and eventually became the chairman of the study section. So that  began a long period of time working with the NIH … So I think I got a certain amount of research training as part of my  exposure to the NIH and to that whole [review] process.”

“[My learning], it’s really by absorption … it’s multiple, accumulative experiences. I tend to read constantly, on a daily basis,  high impact journals in surgery and non surgery. I actually review lots of papers for high impact journals and in it I’m asked  to comment on the design, the rationale, the statistical aspect, the conceptual framework. I view on average three to four  manuscripts a week. And I’m on at least three program committees of three national meetings. I create lots of  abstracts … So it’s really by osmosis. It’s lots of experiences. I tend to go to lots of conferences. I go to research symposia.  And I have never read a book cover to cover but I could tell you that I’ve read numerous manuscripts from first line to last  line.”

“I was very fortunate then because some of the people serving with me on [board] were some of the leading clinical trialists in  the country … So I taught them [subject matter] and they taught me clinical trials design and we sort of worked things  through together. And so that was actually, not really a service thing, but again a really big learning thing. Then that was a  real big payoff.”

“… in those years there was one IRB and that meant we read more than 1000 protocols a year. And after about 15 years I  thought that was enough. But I did learn a lot, you know? After you read 15,000 protocols, you’ve learned quite a lot …”

When describing how they learned about clinical trials, many interviewees articulated that exposure through direct involvement with a trial and the actual experience of going through all the required steps was how they came to understand what trials entail, including the practical realities such as obtaining funding, dealing with the IRB, and managing multicenter studies. “Through that trial I just learned so much about clinical trials … I became much more familiar with all of the IRB issues, change in protocol, protocol deviations. I also attended all of the investigators meetings that took place over the course of five years … So it was a very good learning experience for me … I became a more seasoned investigator throughout the trial.”

Hands-on involvement with a trial was also the first time many were exposed to specific design aspects such as power, randomization, and patient classification/identification, as well as the intricacies of processes such as consent, interaction with regulatory bodies, and how to perform research in a clinical environment. Consequently, in some cases the learning-in-the-field approach lead to a degree of trial-and-error style of progress. “… I feel like through error I’ve got a pretty good grasp on a lot of these things now.” Trial-and-error lessons were cited as a consequence of deficits in educational background. One respondent described their first attempt to design and initiate a trial: “I had no idea what to do. But I took a sample protocol from another study that had been submitted to the IRB and I basically just copied the format and figured it out … I had no idea what I was doing … I just didn’t even know what a protocol looked like. When I came here to the [place] I had never seen a protocol. I didn’t even know what one was.” This same informant believed that “Having junior people who are inexperienced write protocols and do research can probably as much harm as do good.”

Several respondents distinguished investigator-initiated from industry-initiated trials, primarily regarding the role of surgeon investigators and reality that often involvement in industry-initiated trials was less helpful in furthering an understanding of topics such as design and protocol development. “[In an industry-initiated trial], industry develops it [the protocol] and then gives it to the clinical investigator. Do they ask for input from you? No. [Alternatively] an individual [designing an] investigator-initiated trial has very little input from industry, although they may be dependent somewhat if they want to treat patients with new agents … But you know still it’s basically their design and their study.”

Respondents uniformly regarded service-based learning as another highly informative type of experiential learning. Providing service to others by being a member of an IRB or a grant application review committee, reviewing manuscripts, and serving as a departmental reviewer of internal research protocols were regarded as time consuming but extremely valuable on a personal level. The type of informal education gained through exposure to high numbers of protocols, as well as the interactions and debates that occur among (committee) members, was considered by some to be the most important of all the learning experiences. “I think it [service on an IRB] was hugely important … it was a pain in the neck and a ton of work, the number of proposals and the amount of time you put in. But I learned a hell of a lot about clinical research by sitting on the IRB. That’s probably where I learned more about clinical research than anywhere else.” Another respondent said, “I think that that [service on an IRB and on federal grant review panels] really is the basis of my training in clinical trials … clinical trials methodology is evolving, and it’s evolving very rapidly from translational to phase one to phase two trials, to pivotal trials … it’s a field in formation … and my continuing work with my colleagues in the [National Institutes of Health] NIH and the Centers for Disease Control have been very instrumental in developing this.”

Interpersonal learning

Although interpersonal interactions occur in many aspects of research, as discussed by our respondents and referenced here, interpersonal learning describes input from other individuals that directly influences the investigator’s understanding or approach to a given trial (Table 4). Interpersonal learning can take place in direct one-on-one encounters as well as via feedback from anonymous reviewers.

Table 4.

Respondents’ views on interpersonal learning

Learning from experts, peers, and collaborators

“I will give you my opinion on this, and this has been well written about. If you do an experiment in a laboratory where the  results might be dependent on six factors besides the intervention, you can control all of those. You can control the  temperature. You can control whatever it is. The statistics are straightforward####### …. [but] when you start doing research in  humans and the data collection is by humans, the ability for bias to be entered in and the lack of ability, if you don’t  randomize, to ensure control is huge. Once you start doing that level of science it is, in my mind, imperative to have people  who are expertly trained in those areas.”

“[I] went to who I felt was really the leading clinical trialist from a [specialty] standpoint. … without [his/her] help this thing  would never have been resubmitted again. And over the course of really four months of meeting with [him/her] every other  week we were able to design something I thought was really fairly robust.”

“And we had this workshop, talked about all of the issues. You know, what patients should we, well first of all should we have a  clinical trial at all, is the technology mature. The answer to that was a qualified yes by most people. Secondly, what patients  should be included. Thirdly, how should we design the trial, you know, what are the entry criteria. And then of course we had  a biostatistician there, [name], who is the best in the field at doing this kind of randomized trial design. So we went through  all of the issues. And that really clarified things for me, and I think for a number of people also. We knew what the major  issues were. So then it took probably another year to think about the protocol design.”

“The beautiful thing about the VA Cooperative Studies Program … was [the] room full of experts. We had three pharmacists … There  were three biostatisticians. There were representatives from the NIH. There was a statistical representative from the [surgical  specialty] network … and there were two additional [physician specialists], one of them is an experienced NIH trialist and … one  of them is [a] preeminent [specialty] surgeon … So that room was full of brains in terms of running a trial. Really all I had to  come with is the idea and the connection of network because the thing that I brought was the idea and the feasibility … once  they saw that I have a feasible idea, I have the potential sample size, they got excited and they gave me the experts. So I guess  this is a unique situation where I don’t have to find the immediate experts myself and apply for the study. I give the idea. They say  we’ll give you everything provided that the idea and the sample size can be reached.”

[Referring to an NIH clinical trials network] “I was surrounded with lots of bright colleagues, you know … I have the fortune  of being surrounded nationally by people who really bring out the best in me personally and professionally … in that group  are some really outstanding people.”

Learning from mentors

“I had a lot of informal mentoring from people whom I respected a lot … I learned a lot from [individual], reading [his/her]  things and talking …”

“[Name of senior physician] actually was the first faculty member that gave me a case report to write up. And [he/she] told me  how to write it up and edited it and gave it back and gave me feedback, and gave me criticism and told me how to submit a  paper for publication. [He/she also took an interest in my career … I think [he’s/she’s] probably one of the best mentors in  the history of surgery … And I just remember [him/her] saying it’s important that we ask questions and that we answer  questions. The more prospective the questions the better thought out, the better … and this is how it starts … You have a  question. You try to find the answer … [then] start answering the questions the way they should be answered [with  prospective analyses] … when [you have] more resources. That was basically what [he/she] said. And it was very inspiring.”

Learning from the critiques of anonymous reviewers

“Oh I think that’s [external peer review] extremely helpful [when writing and submitting a grant application]. Yes … And the  feedback from that was excellent.”

“I haven’t had a really bad time with the IRB. You know I feel at times they’re a little nitpicky and may miss the point of  things, but if I have a specific concern and don’t necessarily agree with something that they’re pointing out, they’ve been  really accessible in terms of picking up the phone and calling and talking about it.”

“So I want to say that the trial was designed the way it was because of excellent PhD’s [collaborating investigators]. Even with  excellent PhD’s, it was … people that had more expertise at the [name of medical journal] who told us how to better analyze  our data.”

“The input from the national [review] committees was not based at all on [trial] design. It was based on emotion. The most  substantive we got was, ’Since you’re doing this in [name of group] it has no applicability, no generalizability; therefore it  shouldn’t be done.’ There was nothing I could do about that.”

Many respondents described interactions with others in the field as potentially the most important and educational of all experiences, in the context of specific trials as well as career development as a trialist. When asked about valued research resources, one individual stated, “The resources are relationships that get developed … you build those relationships and that’s how you get resources and help to be able to execute clinical trials.” Respondents reported that learning from people often was particularly critical while initiating a trial, when investigators must reach out to local and national experts to answer questions, tackle problems, and/or build a network that would allow a trial to progress. They additionally believed that consulting with peers, collaborators, and content-area experts was valuable while the trial was underway. Ongoing collaboration in a research environment and consultation with experts and specialists in given areas (eg, ethics, statistics) was emphasized at all levels, from defining a research question and setting up/running a feasible trial to analysis of results and publication. These interactions took place in investigator meetings, through independent initiative and contact, and as part of the institutional infrastructure and support system. In talking about the trials they had performed, the accomplished trialists in the sample uniformly devoted considerable time to talking about their collaborating investigators, making it clear that a successful trial, and trialists, depended on excellent collaborators “… it certainly is the case that I did a lot of talking things over with colleagues … There were a lot of logistical problems along the way. We just had to address as we got them. But I had very good people working with me … Certainly I had plenty of conversations with [person] … And so I didn’t do it all myself for sure.”

As for the importance of having a mentor, one respondent told us, “The bottom line is mentorship was a critical part of that early part of my training.”

In addition to the person-to-person interactions described earlier, most interviewees also believed that anonymous reviewers who provided critiques on IRB protocols, grant applications, regulatory applications, and manuscripts played an important part in their education. “So I thought it [grant application] was fairly good so … we submitted it … the comments that came back, while initially hurtful to me personally, were in fact, I think in retrospect, pretty much on the mark … So we tried to allay or mitigate a lot of these risks, risks and concerns of the reviewers. And I think they were all, in retrospect, right on … And I think that’s what I lacked in the initial application was lack of attention to detail of recognizing how it [surgical procedure] might not work, recognizing and measuring how it might not work.” A respondent who was planning a device trial told us, “I made some modifications, many modifications to the protocol based on the FDA. So the FDA actually helped me the most with my protocol because they went through it in excruciating detail and asked fabulous questions and had me change many things … that made it easier to run.”

Recommendations

The recommendations respondents made about how to improve the education and development of surgical trialists can be summarized into 4 directives to aspiring trialists: obtain formal training in the methods and issues of clinical trials; find mentors; establish a network of collaborators; and seek out roles as a participant in all aspects of clinical trials, including their oversight (Table 5). Respondents also observed the key role played by the department chairperson in facilitating the development and progress of their aspiring surgical trialists.

Table 5.

Respondents’ perspectives on how education and career development in surgical trials might be improved

Obtain formal training in clinical trials

“There are a lot of programs in this now that are available now that weren’t available when I was starting my career … there  are even degrees you can get in this or certificates … fellowships you can do as a clinical trialist, and I think it’s worthwhile  to do that. This has become such an incredibly complex field. You can’t just, like I was doing [in my early career] sort of on  the back of the napkin, sketching out the trials and then kind of just doing it with whatever you could find, you know,  resources and so forth. It’s become so complex. And the statistics have become incredibly complex. So you need a knowledge in trial design and you need a knowledge  in statistical design. And I would recommend that they either take courses or they  actually do a year fellowship and you know, get some formal education in this. It will make it much easier because it’s very  painful to start to do these trials on your own and then sort of learn as you go, as you’re doing it. That becomes a very  painful experience.”

“If they [young surgeons or physicians] want to do clinical studies I don’t think anything substitutes for going and getting the  formal training … School of statistics, epidemiology, study design, bias. Formal education … NIH sometime back, funded at  many, many, many, many medical schools short courses in how to do clinical research. Some of them were better than others.  I personally believe that they are too superficial. I believe that you need to go get a level of masters of science, MPH,  whichever you want, but that it is a master’s in epidemiology, statistics and studies if you want to be conversant in doing  good clinical research.”

“… Anyone who is going to be a researcher and have research as a significant part of their career needs formal research  training … People who refuse or don’t feel they have the time and aren’t that committed I try to get them all to take the  [Fundamentals of Clinical Investigation course] … I feel very strongly because I didn’t have that formal training. I think at  least the Fundamentals of Clinical Investigation where they teach the basics of ethics and trial conduct and IRB and animal  care … But I really feel strongly that they should have either an MPH, a masters in clinical investigation or at the very least  take the Fundamentals of Clinical Investigation course.”

“… time is what you make it. If you have the priority then you make the time for it. I think people don’t understand what it  takes to do a human clinical trial. I think you know we don’t learn it in medical school. And then you say, ’Oh, this is not a  big deal. I’m going to test whether this works better than that.’ … [you]start out on the path to get it done and then you’re  first learning all of the hurdles that you have to overcome to get it done. That’s why I think if you’re going to do it you  should have the training up front and then plan after. And the problem, you know, for us as surgical investigators, is there’s  not a lot of infrastructure in place to facilitate it.”

Find mentors

“I think they [potential trialists] need to find a mentor. And for me, I really didn’t have a good mentor. And I just had to do  things kind of on my own. They need to identify a mentor, somebody who has been there, done that, to tell them how to  proceed … don’t be shy to get in touch with people from other institutions and ask them to mentor you. ”

“… what other resource could’ve helped me? ########… .perhaps a mid level to a senior surgeon who could discuss it [the research]  with me in a little greater detail, a little bit more holding my feet to the fire in terms of trial design.”

“… I think the biggest lesson is mentoring is critical at the medical school and college level. If you haven’t got it down by the  time you graduate medical school it’s probably not going to happen. That’s my take, over and over again.”

Establish a network of collaborators

“… try and go to societies or workforces or task forces and see who you might want to collaborate with. And from my  experience in the [professional society for surgical specialty], we have a monthly conference call in which junior faculty bring  up their ideas and bounce them back and forth with experts on the phone, telephone conference. And I can tell you that  there are some really good ideas going back and forth, in terms of where to get the data, how to design the trial, what  sources of funding are there and how to proceed.”

“The American College of Surgeons [offers] a two day symposium for clinical scientists. And that’s one resource for them. The  other resource is that as surgeons there’s lots of professional societal platforms that they can use. And I got to be involved in  this in reverse. Rather than knowing about it and joining it as a junior faculty, I actually got to be known through my  research and I got contacted by our society [of surgical specialty], to become a member of their clinical research trial work  force. So now that I know those things, if I have a junior faculty come to me, I could point him in the right direction and  the resources. I found out that there’s extensive resources. I found out that there’s multiple potential sources of funding but  that junior faculty are not aware of them … I’ve learned them by, I keep saying osmosis. Because I really wish I knew them  right away.”

Participate in all aspects of clinical trials

“I think the best training would be to participate in one [a trial] and just keep your eyes open and see you know what’s  involved, what the work is.”

“Certainly if you’re doing human subjects research you should spend time on the IRB.”

“I recommend that they [aspiring surgical trialists ] have incremental increases in their exposure and skill acquisition … hanging out with people who do this, going to their research conferences, listening to the minutia of why we  make the decisions we do in trial design and conduct and how we do surveillance of ongoing trials … it’s not just designing  a three-page concept. There’s so much more.”

Role of the departmental chairperson

“First of all no one really has the means or time to allocate to any formal training. So we as chiefs have to really provide the  funding for them and the time to do it. When we are developing the leaders of the future, when we are building the career of  young individuals we have to support them and we have to allow them the time to do these kinds of things. And so for chiefs  of department, we end up paying for those courses and we end up allocating the time for them to do it. You know in those  surveys, when we go back, the major barrier to those young individuals for being successful in becoming principal  investigators has been clinical time … most of the time is allocated to clinical work. This is what ends up generating the  most money for a department. I think this is sometimes short-sighted, especially on the side of chairmen of departments  because if a person is successful in research that ends up bringing revenues and recognition to a department as well. So we  have to give them the time and money to do it.”

“To do a clinical trial you have to have considerable resources. Practically the only person who can do it is the chairman of the  department, unless it’s something very simple. I mean a multi-center study is not simple … a smaller clinical trial, like a small drug  trial, yes, that’s simple, and one person could do that. But once you start getting to large numbers I think you need a lot of  resources—either you are the chairman and have the resources, or your chairman is backing you and will give you the resources.”

“I only provide the support for people who are serious and have potential to get funded. We have a limited amount of resources. I  can’t help everyone who wants to do a little chart review project. So if they’re really committed and show me that they’re really  committed to getting the training and potentially moving towards the possibility of submitting for independent funding, I try to use  those resources for those people. So they have to demonstrate to me that they’re serious about what they’re doing.”

“The chairs of departments are also cornered into a situation where they have to generate money and also have an academic  department and be successful academically. And they have to do a balancing act. I think the days when you know each person can  do everything are really over. You’re going to have as a chair of a department identify those people who can really be successful in  research and promote their career by protecting their time and giving them the seed funding. And then you know for those who  have little research inclination or have not been as successful is to direct them into the clinical arena and have them just work  clinically.”

Comments

This study of aspiring and accomplished surgical trialists showed that, for many, professional development as a surgical trialist was a difficult process, with the difficulties in part stemming from a learning process that was haphazard, inefficient, and therefore personally costly, and, for some, marked by mistakes that might have been avoided.¹² The success of almost all these trialists came only after intense persistence at every step, much personal sacrifice and commitment, and learning experiences that often taught them what not to do instead of what to do. Our respondents told us that 4 types of educational and developmental resources played constructive roles in their evolution as trialists: formal education in clinical trials, written materials relating to clinical trials, experiences with actual trials, and interpersonal interactions with others involved in clinical trials. The degree to which individual trialists used these varied, as did perceptions of their relative usefulness. These findings must be interpreted in light of our purposive sample of 15 respondents. Although we tried to and did achieve a diverse sample, and continued accruing until we perceived no new information was being provided, our sampling may have left out some types of surgical trialists and their insights.

Regarding formal education in research, this was a highly educated sample. Nearly half obtained some kind of formal education in clinical trials beyond their doctoral degrees. However, most of our respondents expressed strong regret at the inadequacies of their didactic instruction that became apparent as they tried to conduct clinical trials. In fact, outside of mandatory human subjects credentialing requirements, the ability to obtain formal education in clinical trials appeared to be a haphazard process wherein individuals were challenged to find available programs or the time to participate in them.¹³ Several respondents mentioned the importance of short courses in clinical trials offered by organizations, and other respondents mentioned the importance, although as only an educational starting point, of training in clinical research available through the National Institutes of Health (NIH) K30 Clinical Research Curriculum Award.¹⁴ Only 51 institutions have received funding for this endeavor. One missed opportunity for formal preparation in surgical trials is the research year(s) during surgical training: 9 of the 11 surgeons in our sample had invested in such years, but their experiences were solely in the laboratory.¹⁵ Although every clinical trial poses some unique challenges, formal education in the methods of clinical trials equips a trialist with a foundation on how best to approach and overcome them.

We began this study because we were curious about the extent to which surgical trialists used the clinical trial documents and guidance published by regulatory and professional organizations that we have uncovered.^7,8 The short answer is, not much. Only 3 respondents even mentioned any of those documents. This seems unfortunate, given the wealth of information these documents contain on how to perform ethically sound and methodologically rigorous clinical trials and how to report them. We speculate that because a systematic approach is lacking on the professional formation of surgical trialists, most will encounter those documents, even those focusing on specific areas of interest for surgical trials (eg, CONSORT guidance for clinical trials of nonpharmacologic treatments)^16,17 by accident, if at all.

Overall, the use of literature as an instructional resource was limited to scientific articles used to answer clinical questions. Although one respondent told us that voracious reading of published clinical trials was how he/she schooled him/herself on the methods of clinical trials, published literature largely was disregarded for matters such as study design or ethical and methodologic considerations of human subjects research. In fact, with the exception of other trial protocols and published reports of trials, our respondents generally found written materials to be a frustrating resource to try to use.

In contrast, our respondents believed that experiential learning and learning in the course of interpersonal interactions were critically important to their professional development and success as a trialist. It was highly emphasized that without a strong supporting research infrastructure, whether institutionally based, through a professional society, or an individually constructed network/team of experts in various areas, surgeons are unlikely to succeed at investigator-initiated trials.^18,19 As many interviewees pointed out, the complex nature of research design, statistical analysis, and logistical coordination often prove beyond even the most capable individual to follow through on an idea. In contrast to industry-initiated trials, which one respondent described as “where you’re handed the protocol, you search for the patients and go from there,” surgeons who want to evaluate an intervention or procedure need institutional support and networking resources to designing the trial, obtain IRB approval, find funding, conduct the trial, handle the data, and write up the report.

In addition, although trialists found experiential and interpersonal interactions to be some of the most important education on how to develop and execute a trial, in most cases these were the least structured resources. The few interviewees who did have an institutional infrastructure that provided the necessary “room full of brains” (as one respondent called it) believed very strongly about the benefit of this arrangement. Others who independently sought out and assembled a network often used terms such as “lucky” and “fortunate” when talking about finding others willing and able to support their projects.

Overall, our findings show a remarkably unsystematic process by which highly motivated individuals struggled to find the information and resources necessary to develop competence as trialists and to mount scientifically and ethically sound surgical trials. The education process was cumulative and involved varying degrees of formal and informal instruction, exposure, interpersonal interaction, and hands-on experience. Although on some level this is representative of the field and its nature as a continuous learning process, it also is indicative of the insufficient structural and educational systems in place and available to surgeons. Many respondents described an initial experience of feeling largely unprepared and somewhat “surprised” by obstacles and their own insufficient knowledge base, and expressed how this compounded the many obstacles inherent in attempting to initiate and run a surgical trial.

Although the education and career development of surgical trialists always will be a multifaceted process that should be tailored to the needs of the individual, the haphazard and unstructured nature of the current system is only adding to the already numerous challenges faced by surgical trialists.^20–23 As human subjects research becomes increasingly complex and ever more regulated, surgical trialists without an adequate educational background and support system likely will find themselves unable to overcome the numerous challenges they will face. Those challenges include competing successfully for external support for trials in an era of little growth in the amount of potential funds but substantial growth in the number of highly trained competitors from fields outside of surgery. To improve the current surgical trials workforce, the surgical profession must address the availability as well as content of educational resources, and focus on practical instruction.²⁴ Strengthening the surgical trials workforce is critical in improving the quality of scientific evidence for invasive therapeutic interventions and their adjuncts. The recommendations made by those we interviewed, formal education, mentoring, networking, participating in all aspects of trials, and facilitation by department chairs, indicate that the responsibility for improving the quality of surgical scientific evidence lies not only in the hands of investigators, but also to a large degree in the hands of departmental, institutional, and external programs.²⁵ Without efforts on the part of all those involved, neither the quality nor quantity of surgical clinical trials have much hope of forward progress, especially as time and other pressures are increasingly exerted on young surgeons.