Successful Implementation of a Novel Internet Hybrid Surgery Curriculum

Abstract

Background:

The internet CD-ROM thoracic surgery (TS) e-learning system was implemented in 2001 as a prospective randomized trial testing resident acceptance and educational impact of a unique web-based curriculum system on prematriculated TS residents. The Prerequisite Curriculum (PRC) contains 75 segments organized with textbook and case-based navigational systems.

Methods:

Web-based technology tracked the PRC use for each resident. Of 142 residents, 138 thoracic surgery residents matching in 2001 for 2002 matriculation participated in a prospective randomized trial comparing the PRC system to a control group. Two sets of in-training exams, as well as resident and faculty knowledge/performance surveys, were used from July 2001 through January 2004 for ongoing, blinded multidimensional evaluation.

Results:

Most residents (55/69) responded to the written prematriculation surveys and indicated they used the PRC (43/55), averaging 1.45 hours weekly. The PRC was rated as easy to use (8.3/10), a valuable study guide (7.7/10), and superior to traditional texts and journals for preresidency preparation (7.9/10). Web-based tracking revealed that 47/69 actually used the PRC. Sessions averaged 23.3 minutes with an average of 148 sessions over the prematriculation year. The in-training exam performance when evaluated at 1 and 9 months into the TS residency revealed a positive correlation between examination performance and PRC use. After TS residency matriculation, the self-evaluated knowledge and performance satisfaction scores were superior among PRC users in all categories. Simultaneous TS faculty evaluations of the same resident groups demonstrated smaller, but significant group differences.

Conclusion:

The implementation of the TS PRC has been exciting and successful. Future multidisciplinary curricular progress will hopefully continue to build upon this e-learning strategy.

An internet CD-ROM e-learning system was implemented as a prospective randomized trial testing resident acceptance and educational impact on matched prematriculated Thoracic Surgery residents. Follow-up surveys of knowledge and performance, web-based tracking of utilization, and the ABTS ITE scores were among 8 modalities used to suggest successful early outcomes of this innovative teaching tool during the ensuing 3 years.

Thoracic surgery resident education continues to undergo considerable change. The cognitive and technical content of our educational process, the structure of our educational process, and the venue/timing of the delivery of the educational process are all evolving.¹ Although the formal thoracic surgical residency education period spans 2 or 3 years, we continue to rely upon a significant amount of didactic, clinical, and technical skills that have been learned by the resident prior to initiating the formal thoracic surgery residency process. The rapid growth of fact-based information and clinical skills pertaining to this surgical discipline has resulted in a significant expansion of the amount of material we must successfully transmit to our residents.^2–5 There is an increasingly severe shortage of residency time and, therefore, an increasing reliance on the transmission of a solid foundation in fundamental surgical skills and knowledge to our residents prior to the very first day of their thoracic surgery residency matriculation. The ongoing changes in the work hour time available for resident education in the hospital or clinical setting and the distribution of those hours during the typical residency workweek has further increased the challenges of transmitting increasingly large quantities of important cognitive information,^6,7 as has the shift to ambulatory procedures and other changes further separating residents from clinical and didactic learning opportunities.

PURPOSE

The Thoracic Surgery Directors Association (TSDA) Prerequisite Curriculum was structured to maintain an inclusive outline and a robust catalog of factual knowledge that would be “optimally required” for residents to master prior to initiation of their Thoracic Surgery residency. As a secondary goal, the TSDA focused on the development of an innovative methodology to deliver that knowledge in such a way that it would be highly efficient, impart enduring knowledge, and become easily integrated into the didactic spectrum of our residency education programs. As a final goal, the curriculum content would be immediately useful and later serve residents as a permanent reference before, during, and after the completion of formal thoracic surgery residency.⁸

To do so, it was necessary to analyze the current structure of thoracic surgery education and determine a useful division of the didactic material that would be transmitted before as opposed to during the formal residency program.^9–17 The educational process involves subject material that can be categorized into the surgical and nonsurgical components of education. We have traditionally taught both the technical and cognitive aspects of the thoracic surgery curriculum in a highly synchronized, “lock-step” fashion. Removing a discrete component from this long standing, traditional and highly synchronized process was to be the first key step for this project.

The TSDA has provided a requisite core curriculum outline for the technical, clinical, and didactic material that is to be taught during the entire span of the thoracic surgery residency.^18,19 It is the intent of the prerequisite curriculum project to focus on the didactic aspects of that information that we believe residents should master before beginning a thoracic surgery residency. The matriculating residents will then be theoretically well positioned to begin to build upon this knowledge and acquire new knowledge traditionally taught and self discovered during the thoracic surgery residency period. In addition, it was anticipated that our residents would be most receptive to the technical and clinical aspects of their thoracic surgery residency if their cognitive base was solid. This would “set the stage” for their residency and life long learning.

The time interval dedicated to thoracic surgery residency, although under intense discussion, is currently well defined. It was the hope that the prerequisite project would address the portion of the didactic component that could be initiated before the completion of a general surgery residency and contain specific content relevant to both general surgery and thoracic surgery residency education. It would extend from the time that the resident were successfully matched into a thoracic surgery residency at the end of their fourth year until they matriculate in a Thoracic Surgery residency, approximately 12 months later.

DEVELOPMENT

To accomplish this goal, the Thoracic Surgery Directors Association created a Prerequisite Curriculum Committee that began its work in September 1998. The Committee goals were (1) to develop the content outline of a prerequisite curriculum and (2) then develop an implementation plan and associated methodology to evaluate the effectiveness of the content and of the implementation of a prerequisite curriculum during the course of several cycles of residents for a minimum of 5 years. A secondary goal was to develop a CD-ROM Internet high technology hybrid educational product that would allow the resident to study this curriculum employing the most modern techniques of electronic-based education or e-learning.

The curriculum project contains a total of 75 topics known as modules or segments which are broken into 13 textbook-like chapters, known as sections (Table 1). These 13 chapter-like sections span the most basic areas of our core residency curriculum. It also includes several supplemental topics to facilitate ongoing learning and expertise development in all of the traditional and nontraditional core competencies. Many of these segments, and indeed several of the sections, are highly germane to ongoing study during the chief residency general surgery year and in preparation for American Board of Surgery qualification and certification examinations.⁸

TABLE 1. The 13 Sections and Total Completed Study Session Views for Each of the 13 Prerequisite Thoracic Surgery Didactic Prerequisite Curriculum (PRC) Sections in ′Textbook Navigation’ Sequential Order

The textbook sections were divided by the 6 editors of the Prerequisite Curriculum Committee coordinating the efforts of more than 60 participating authors. These authors include not only thoracic surgeons, but adult and pediatric cardiologists, anesthesiologists, bioethicists, pulmonologists, intensivists, statisticians, radiologists, as well as many other specialists, who not only are extremely knowledgeable in their field, but are also highly experienced educators and as such can easily articulate their subject material to transmit somewhat “dry” didactic subject material in an engaging electronic audiovisual format.

Each of the 75 segments of the 13 sections was built around a 12- to 34-minute audiovisual presentation narrated by the individual segment author. Relevant media, including video photography, radiographic scans, images, and other material, were included. Each segment also contains a 2- to 6-page summary document as well as a set of frequently asked questions (and answers), a bibliography, a set of relevant Internet linkages, and a multiple-choice /true-false, self-assessment examination for each resident to complete before being able to consider the segment as “done” (Table 2).

TABLE 2. The 8 Components and File Locations of Each of the 75 Segments of the Prerequisite Thoracic Surgery Didactic Curriculum

In addition, each segment has multiple evaluation questions relevant to the segment just completed. Responses to the numerical and free text segment critiques are periodically provided automatically to the prerequisite curriculum editorial board to assess the content, format, and quality of each of the individual segments of the curriculum. It was the responsibility of the Prerequisite Curriculum Editorial Board to assemble these segment critiques and continually review these 75 segments, including each of the 8 segment components that constitute the completed segments or e-learning modules for possible immediate modification or exclusion from the current or future versions of the curriculum.

The CD-ROM Internet hybrid product itself represents a high technology marriage between the internet and CD-based technology. All of the summaries, self-assessment material, references, internet linkages, frequently asked questions, and segment critiques are stored on web-based servers maintained by the TSDA. In contrast, the high band with video and audio material is, for the most part, stored on a set of serialized CD-ROMs. This was done to minimize utilization loading type delays and associated streaming video types of user frustration. A unique user security system was developed, tested, and implemented to securely identify the user and prevent ‘sharing’ of content or user identities.

The examination and critique responses, as well as the time investment pattern of each of the learners, were stored simultaneously on the TSDA server. For individual residents to successfully interact with the curriculum, the CD-ROM must be installed on their computer, and the computer initially connected to the Internet. Periodically the user needs to reconnect to the Internet to update the material and to be able to access the web-based portion of the curriculum. The web-based software tracks the amount of time that the residents spend on each of the segments of the curriculum, their performance on self-examinations, and their responses to all of the critique sections. Curriculum content and format updates are automatically distributed through the Internet. The user education and ongoing support for technical questions are addressed in a similar fashion.

One of the unique and key features of this curriculum is that there are 4 different ways to navigate through the entire curriculum content (Table 3). The textbook-based navigation system is similar to a table of contents with numerous sections and subsections related to the breath and depth of the prerequisite curriculum. A second means of navigation is the case-based system in which 12 clinical surgical cases together make up 75 segments (Table 4). By managing the 12 individual patients under consideration, the residents would be able to acquire the same amount of material as if they had worked through the textbook-based section in a sequential fashion. There is also a media-based navigation system that allows the users to specifically select a video clip, audio segment, pathology slide, x-ray, angiogram and so forth, based upon a specific disease state or organ system they are interested in studying. Finally, there is a subject or topic search methodology through which specific focused education materials, are available, including the slide set portions, case-based portions, and media portion, by simply searching for single or multiple keywords. Any one of the segments and segment subcomponents can be located and learned or relearned in any sequence and at any time. Once a segment is completed, the users receive credit for the segment and are informed as such should they revisit the segments at a later time.

TABLE 3. The 4 Navigation Systems and Relative Use During the Initial Prematriculation Year to Complete the Required Segments of the Prerequisite Curriculum Material

TABLE 4. The Topic and Case Sequence of the 4 Subject Areas and the 12 Clinical Cases of the Prerequisite Thoracic Surgery Didactic Curriculum “Case-Based Navigation” System

IMPLEMENTATION

In June 2001, the Thoracic Surgery Residency Match results for residents matriculating in 2002 were released. A letter describing the Prerequisite Curriculum Project was mailed to all of the matched residents, to their thoracic surgery program directors, and to their general surgery program directors. It included a description of the project and requested the residents to sign an informed consent allowing them to be prospectively randomized into one of 2 groups, those receiving the full CD-ROM Internet hybrid curriculum and those receiving only an outline of the content of the curriculum, appropriate references, but no educational content. Great care was taken to inform all of the thoracic surgery residents and general surgery program directors of their residents’ participation in the program but not of their randomization status. Each resident also agreed not to share their randomization status with his/her faculty, current program director, and future program director or with his/her coresidents. The TSDA reassured all participating residents that his/her randomization status and performance/utilization parameters would not be identified with them in any way, and as such, never shared with their faculty, program directors, or used in the credentialing, promotion or certification process.

In August of 2001, the residents agreeing to participate (having signed the informed consent) were randomized such that 50% received the CD-ROM Internet hybrid curriculum and the other 50% received the curriculum outline. Of a total of 142 residents asked to participate in the process, 138 agreed to participate and 69 were randomized into each group. They were instructed to proceed at a pace of approximately one section (approximately 7 segments) per month, allowing all of the sections to be completed in the course of the upcoming prematriculation year. They were informed that their time investment in each of the sections would be tracked on the Internet as well as their critiques of the individual segments. The 8 intended evaluation and follow-up modalities were described along with the associated projected time lines for the implementation and subsequent and subsequent evaluation.

During the ensuing 24 months after the curriculum was distributed, the time investment patterns of the residents have been tracked on a weekly basis. There have been follow-up phone calls and written survey questionnaires for all of the randomized residents on a four-month basis requesting specific information regarding the ease of use, overall evaluation and in particular, any technical problems or suggestions they might have during the time period from their randomization until their matriculation in the thoracic surgery residency program in July of 2002.

The evaluation of the 2 resident groups has been extensive (Table 5). All of the residents matriculating in July 2002 were asked to take the American Board of Thoracic Surgery In-Training Examination (ITE) in August 2002 reflecting their didactic base of knowledge at a time 4 weeks after they began their thoracic surgery residency. They also took the (different) ensuing in-training examination given by the American Board of Thoracic Surgery in April 2003. An opportunity for additional follow-up information concerning the transition from prerequisite to core curriculum, as well as performance on subsequent in-training examinations and ultimately the American Board of Thoracic Surgery qualifying and certifying examinations have been structured. Evaluation techniques were also developed and implemented during the residency to assess other areas of impact of the prerequisite curriculum on the thoracic surgery residency and thereafter. In particular, psychologic and performance parameters, comfort levels, and the overall satisfaction of the residents were assessed.

TABLE 5. Outcome Evaluation Parameters to be Used for the Prospective Trial After the Randomized Implementation of the Prerequisite Curriculum CD ROM Internet Project

RESULTS

Of the 192 resident surveys mailed to 69 residents who received the Prerequisite CD-ROM Curriculum 100 (64%) responded during the 3 mailings in December 2001, March 2002 and June 2002 to the thoracic surgery preresidency survey (Table 6). Of these residents, 78% indicated that they used the Prerequisite Curriculum, with wide variation averaging between 0.7 and 2.2 hours per week during the prerequisite year. Utilization markedly increased following the general surgery in-training examination in January of that year. These residents indicated that the Prerequisite Curriculum was easy to use (8.3/10 when evaluated on a 10-point scale such that 10 was most useful and 1 was least useful). When asked whether it was a valuable study guide the residents rated it as 7.7/10. When asked whether it was superior to traditional texts and journals as a preferred method to prepare for their Thoracic Surgery residency, the residents rated the prerequisite curriculum as 7.9/10. When asked about the usefulness of the visual/verbal content versus the written content, the residents uniformly favored the ease of use and educational impact of the visual/verbal materials.

TABLE 6. Summary of Prematriculation Resident Written Survey Information Evaluated 11, 8, and 5 Months After Receipt of the TSDA Prerequisite Curriculum CD-ROM Internet Product Prior to TS Residency Matriculation

Internet tracking results were compiled for all residents using the curriculum (Table 7). The utilization patterns varied considerably in average and on a resident-by-resident basis from those reported by the residents in the survey tools they completed. A total of 47 of 64 residents receiving the CD-ROM set actually used the prerequisite curriculum more than once and only 31 of the 69 residents receiving the curriculum used the product for more than 20 sessions. These 47 residents had a total of 3161 educational sessions averaging 19.6 to 23.3 minutes each, with an average of 148 sessions per resident over the prerequisite year. This was measured to be an average of 1.44 hours per week for each resident using the curriculum.

TABLE 7. Summary of Prematriculation Internet Tracking Information Measured Immediately Before Residency Matriculation After Prospective Randomization of the TSDA Prerequisite Curriculum CD-ROM Internet Product Approximately 10 Months Previously

Most of the curricular work (92%) was done using the textbook navigation system, the remainder (8%) done using the case-based navigation system. A total of 44% of the sessions were completed between 8:00 am and 4:00 pm, while 51% were completed between 4:00 pm and midnight.

When the raw scores of the American Board of Thoracic Surgery In-Training examination (ITE) administered 4 weeks into the residency (August 2002) were compared among the 2 randomized groups of residents there was no difference between those residents randomized to receive a CDR set and those randomized to receive the curriculum the outline. There was however a positive correlation between examination performance and the total sessions of the Prerequisite Curriculum used among those residents, using the curricular materials more than twenty times during the prematriculation interval (Fig. 1). This trend persisted, but was less prominent when the 2 groups of residents were compared 9 months after initiation of the thoracic surgery residency.

FIGURE 1. American Board of Thoracic Surgery (ABTS) August 2002 In Training examination performance for all residents matriculating in July 2003 using the PRC 20 or more times during the year before the examination.

The results of the resident critiques were reviewed for 845 of the 3202 completed segments (Table 8). In addition to free text comments, the residents were asked: (1) How useful/relevant do you believe that the segment is? (2) Did the segment provide a solid foundation for additional clinical and didactic study? and (3) What was the overall quality of the format and presentation of the segment? The choices offered ranged from not useful/low quality to very useful/very high quality. Ninety-two percent of the segment critiques were in the useful/high quality or very useful/very high quality range. The detailed break down of this information, in conjunction with the utilization/completion data for each segment and the free text comments, is of great importance to the ongoing editorial process for the prerequisite curriculum.

TABLE 8. Summary of 856 Prematriculation Resident Evaluations of All of the 75 Segments of the Prerequisite Curriculum (PRC) Evaluated During the Year Prior to TS Residency Matriculation

Postmatriculation program director resident surveys (n = 250) and resident self-evaluation surveys (n = 138) were administered and analyzed 3 months into the first residency year (Table 9). These written surveys sought to evaluate the 13 specific curricular sections of the basic prerequisite curriculum covered in knowledge, teaching, and application of knowledge. Twelve areas of resident “comfort and satisfaction” with their educational process were also surveyed on this instrument. The groups were distinguished based upon randomization status and upon actual utilization of the PRC for twenty or more sessions during the prerequisite year. Each of these comparisons was made using a paired t-test and reported as significant when P < 0.05. On all of the resident self-assessment survey parameters there was a significant difference (P < 0.05) favoring those residents receiving and using the PRC both in knowledge, teaching and application of knowledge and in all of the comfort/satisfaction parameters.

TABLE 9. Summary of Faculty and Resident (Self-Assessed) Overall Knowledge, Teaching Knowledge, Application of Knowledge, and Resident Performance/satisfaction Survey Responses for Residents Matriculating in July 2002 Determined 3 Months After Matriculation on an Expectation scale of 1–5

On the 3-month postmatriculation surveys, the program directors reported smaller but significant differences in overall knowledge and application of knowledge favoring those residents randomized to receive the PRC. No significant differences were seen in the teaching and in the comfort/satisfaction parameters when the groups were compared by their program directors. The program director surveys were blinded as to the randomization and utilization data for the residents involved. Interestingly, the program director scores evaluated on an identical scale were statistically higher on average in each of the fifty-one categories of knowledge and performance than the resident's self-assessment of the same category of knowledge, particularly for those residents not receiving the PRC.

LIMITATIONS

This study is limited in that it is predominantly descriptive in nature given the relatively small total number of randomized residents involved and the even smaller number of residents actually using the curriculum materials in what we believe to be a meaningful way. Although it was not possible to scientifically define meaningful utilization, the benchmark of twenty total sessions or an average of one 20-minute session every other week prior to matriculation was used to facilitate the analysis. Had a larger number of sessions been required to be deemed significant, such as half of the sessions, the total number of residents available for analysis in this group would have been significantly smaller yet. The lack of statistical power was anticipated given these small numbers and the imprecise parameters available to measure impact of this type of curricular implementation.

The educational outcome assessment tools that were used do not mirror the content that was delivered, nor do they separate the outcomes of a highly motivated learner from a less motivated learner. In particular, the American Board of Thoracic Surgery ITE has a scope far broader than those topics covered in the PRC, as it is designed to cover the core curriculum as well as all topics within the prerequisite curriculum. Attempts to look at responses by randomized residents to a specific ITE question set specifically relevant to those topics covered within the Prerequisite Curriculum and not the entire examination was attempted and rapidly became logistically impossible.

Extending the study to additional generations of residents matriculating in subsequent years was considered but was felt to deprive matching residents of a potentially highly effective learning tool. Further evaluation is of little potential for better comparative data, as the teaching tool will become more widely available and will likely be “shared” among residents and program directors. There is no way to accurately measure the amount of sharing that occurred prior to and following the matriculation date described in this prospective randomized study, thus contaminating the 2 groups. There was a true element of individual and group competition between the 2 groups of residents, thus skewing the performance of the control group away from historical baselines. In addition, those residents in the control group were promised the full curriculum upon matriculation as a condition for participation.

The difference between the resident survey data and tracking of utilization measured objectively on the web underscores the benefits of objective measurements and the associated inherent challenges in self-reported surveys. Analysis of these differences revealed substantially less utilization (sessions and duration) than initially anticipated and then self reported on many of the survey tools. It is unclear how the TSDA expectation that the curriculum was to be completed prior to residency matriculation influenced survey responses.

DISCUSSION

The Thoracic Surgery Directors Association Prerequisite Curriculum Committee has successfully developed and implemented content for a didactic curriculum to be mastered by the residents before their matriculation into thoracic surgery residency programs. In addition, the Committee developed an innovative electronic format consisting of a CD-ROM Internet hybrid to deliver this curriculum material. By the use of the serialized CD-ROM Internet hybrid, it is possible to store relatively dense and high bandwidth portions of the curriculum, including video and audio materials on the CD-ROM and yet allow constant updating and interaction with other portions of the curriculum. In addition, it has been feasible to track the performance and utilization by the residents during the course of the residency program. Studying the relationships between the utilization of the curriculum, performance on standardized examinations, as well as resident self-assessments and program directors’ assessments reveals that the delivery system and content were well received, and trends favored improved outcomes related to higher utilization of the materials.

It is anticipated that the process of reviewing content and developing new content of the Prerequisite Curriculum will allow the residents and program directors to continually focus on the subject material deemed necessary for successful initiation of the thoracic surgery residency. Providing much of this material outline (as opposed to on CR-ROM) facilitates rapid and dynamic management of continually updating content and technology.

Given the recent changes that have occurred regarding resident work hours, the ability to deliver high quality educational didactic material, clinical material, and hopefully technical materials through a distance learning system and to track utilization and performance becomes even more appealing. This form of technology has broad-based applicability as a highly specialized learning management system focused on distance learning in a cohort of residents not directly under the watchful eye and gentle hand of their thoracic surgery program director.

Although electronic-based education has been available for many years, a prospective randomized study comparing it with traditional textbook-based learning in thoracic surgery is novel. Multiple attempts have been made to implement web-based, CD-ROM based, and integrated educational tools in many specialties with variable results.^20–36 It is our anticipation that successful completion of this project will not only allow for the use of an innovative, highly effective means of education for our residents, but may in turn become broadly attractive to other types of educational projects within thoracic surgery education particularly in the venues of life-long learning and maintenance of competency. This may also be suitable for other types of educational projects in the postgraduate education universe and other venues of nonmedical education as well. The conduct of prospective randomized scientific studies monitoring the impact of curricular projects, as well as the measurement of acceptance of the curricular materials by the residents and program directors is a highly desirable implementation model and will hopefully be continually scrutinized and improved by educators.

ACKNOWLEDGMENTS

This project would not be possible without the extensive assistance of numerous individuals, including the efforts of the staff and leadership of the Thoracic Surgery Directors Association and particularly hose of Ms. Martha Klapp. These individuals were critical to the collection and processing of the masses of information required to assemble this content and develop the program. The 60 segment authors, as well as the technical efforts Dr. Anthony Mercando and Mr. Arismendy Ovalle of Amadeus Multimedia, were also critical to the completion and the ongoing monitoring of this project. The TSDA prerequisite curriculum committee, including Drs. Malcolm Decamp Jr., Verdi J. Disesa, Eugene A. Grassi, Thomas L. Spray, Richard I. White, Douglas E. Wood, and Joseph B. Zwischenberger, were central to the maturation of the content of this curriculum, as well as to the collection and editing of all its sections and segments. They have also been intimately involved in the ongoing segment evaluation and the planning for the next generation as the PRC Editorial Board. This project has been supported in part by unrestricted educational grants from Ethicon Endosurgery Inc. (Cincinnati, OH), and by Edwards Lifesciences, LLC (Irvine, CA). Without the generosity of our corporate supporters, this project would not have been possible.

Discussions

Dr. Carlos A. Pellegrini (Seattle, Washington): When one first reads this paper one is immediately drawn to the results of this prospective randomized double-blind comparison of a group of residents given a fancy new tool with a group of residents who did not have access to it. And when one looks at the results when objectively measured with the ABSITE test, one does not see a remarkable difference that one would expect, and one is tempted at first pass to consider that perhaps the authors’ efforts were educationally trivial. Do not fall into that trap. Nothing could be further from the truth. In fact, I submit to you that the Thoracic Surgery Director's Association has made one of the most significant contributions to surgical education, not just for thoracic surgery but for all of surgery.

First, in this paper the TSDA defines a threshold of knowledge that residents entering their training are expected to have. The threshold is logical and intended to prepare incoming residents with the tools necessary to be safe, effective, and to fully participate from the start in an advanced training program. Secondly, they developed a standard curriculum, which thus applies uniformly to all programs. Third, they set the expectation that those who want to join their specialty must adhere to the principle of life-long learning.

If they had done nothing else, I think they would have already gained the respect of the surgical community. But they did a lot more than that. And what they did is that they developed an innovative methodology to deliver this knowledge in an efficient and comprehensive way. And to top all of that, they then studied the implementation process so that they were able to determine how frequently the new tool was used, for how long, by whom, and whether it had an impact or not.

I confess that I was a little bit surprised to see that only three-quarters of the residents who received this new tool even installed it in their computer, and that less than half of those randomized to it used for more than 20 sessions. I was equally surprised by the seemingly small amount of time (1.4 hours a week) spent studying.

There are of course many potential reasons for this: those who received the tool may prefer other ways of acquiring knowledge, the topics may have been too trivial for a fourth-year resident in general surgery, or, as I believe is the case, they might have been completely consumed with their general surgery training and chose to use their little free time in some other way. In any case, I am surprised, and perhaps disappointed, that this tool, clearly directed to the new generation of computer-savvy residents was a bit under-utilized.

However, I am optimistic that this is the way of the future. Those who used it felt it was useful, and when self-evaluated or evaluated through the eyes of their attendings, they seemed to be better equipped than those who were not. So I would not be dissuaded by some of the negative findings of the study. Instead, I predict in the years to come we are going to look at this study as a landmark piece. I also predict that other director's associations are going to follow through.

Thus, my real question to the authors is: Could they share with us what they believe are the real costs in terms of money, energy, time, et cetera? Using what they learned with this study, could they share with us what is it that a surgical association should do to produce this kind of product? And lastly, would they tell us what is the cost-benefit ratio as they see it now?

Dr. Jeffrey P. Gold (Bronx, New York): Thank you, Dr. Pellegrini for your insightful comments, your excellent questions, and for your legendary educational leadership.

This curriculum took 5 years to develop. A much smaller percentage of the time was invested in creating software and testing software than it was in collecting high quality content and organizing that content from members of our specialty and from other specialties.

The financial costs at the time we did this, starting in 1996, were high. We invested approximately $340,000. Fortunately, this was all provided through non-restricted educational grants to our corporate sponsors. Were you to do this all over again today, you could probably do it for, I would say, 40% of that number, given the advanced technology and the far smaller costs necessary for multimedia presentation.

A large amount of time has been invested by the voluntary physician leadership of the Thoracic Surgery Director's Association. The Prerequisite Curriculum Committee has worked long and hard, as has the leadership of the TSDA, over several generations to turn this into a usable reality.

Your point is well taken in that e-Learning is very much like the early days of laparoscopic surgery. I can recall back when I was a general surgery resident that it was slow, awkward, and it was expensive. It was highly doubted. Looking back on that, nobody would say that about the current techniques. I feel confident, as Dr. Pellegrini articulated so elegantly, that nobody is going to say that about e-Learning as we look back on this in the future.

This project represents just one of the small and early first steps. I invite you all to look at the curriculum itself, which is available as public domain through the Thoracic Surgery Director's Association Web site.

Dr. William A. Baumgartner (Baltimore, Maryland): I would like to congratulate Dr. Gold and his colleagues on an excellent presentation and a novel approach to thoracic surgery education.

In a somewhat general medical school environment where the priority of education is usually last behind clinical care and research, this work is particularly important. It demonstrates how effectively research can be applied to education.

In the current regulatory environment of decreasing work hours, the use of innovative educational techniques will become increasingly important for the successful training of residents. The other far-reaching implication of this work is its application to practicing thoracic surgeons as they continue their ongoing education as part of maintenance of certification.

The easy ability to use portions of this TSDA curriculum for future educational content demonstrates the foresightedness of this work. This content has been written in a standard language of e-Learning, which allows other future content to be developed without redundancy. I can best illustrate this by example of a model train. Each car represents different areas of content. But each car has to conform to a standard gauge, which then allows interchange and substitution of new educational content without duplication. Dr. Gold, would you please comment on this further aspect of your current work?

Increasing research and education will have to be a priority as all residency programs will continue to change. I congratulate you and your group for being leaders in this area and thank the American Surgical Association for the opportunity to comment.

Dr. Jeffrey P. Gold (Bronx, New York): Thank you, Dr. Baumgartner. Your comments are also very timely.

The use of a common platform to move educational content across the Internet is going to be the final common denominator to allow us to take content from many different specialties and many different levels of education and share them. It is just going to be too time-consuming for every organization and too expensive for every specialty to develop their own standards of e-Learning, and therefore a standard that is commonly known as the Shareable Content Object Model, or SCORM, model, is the one that we have used. Once the standards are set, we can encourage content delivery systems, we can develop learning portfolios, and we can develop very robust banks of quality medical content.

Dr. John R. Benfield (Los Angeles, California): Time is a major factor in all education programs, and thoracic surgery is no exception. For example, thoracic surgery training program directors struggle with the competition for residents’ time during the first year of thoracic surgery residency that results from the residents studying for the examination of the American Board of Surgery. Other examples of the competition for time are the issue of how much time should be devoted to general surgery and how should time be divided between cardiac surgery and general thoracic surgery? Certainly, the 80-hour workweek issue is another example. The e-Learning method project is a method that can make thoracic surgery education more time efficient.

I would simply like to recall a little more history than Dr. Gold had time to do. The Thoracic Surgery Director's Association, or TSDA, published its first comprehensive curriculum in 1995 when Gordon Murray was president.

In 1996 when I had the privilege of the presidency, TSDA had its first retreat of 91 program directors, 53 associate members, and 12 Canadian educators, for the purpose of studying the prerequisite education for thoracic surgeons. Jordan J. Cohen, the president of the American Association of Medical Colleges, gave the keynote address, in which he advised us not to compromise quality while adapting to the need for increased efficiency and financial constraints. Doug Mathisen, a subsequent TSDA president and one of Dr. Gold's co-authors, led 19 people who focused upon the prerequisite curriculum and identified eight mandatory, 6 desired, and 4 optional categories of knowledge and skills.

I want to congratulate the authors, Dr. Gold, and TSDA for implementing the innovative and time-saving e-method. Dr. Gold, you have told us nicely about the experience with e-Learning and the prerequisite time. Where does the TSDA stand with regard to using e-Learning during thoracic residency and how about adapting the method for continuing education?

Dr. Jeffrey P. Gold (Bronx, New York): Thank you very much for your comments and also for your educational leadership.

The TSDA is quite active in using this technology for its core curriculum, and indeed next week in Toronto at our annual American Association of Thoracic Surgeons meeting, the core curriculum educational modules will be rolled out and made available to all residents. These learning modules are also available on our web site. We will have reached our quest, at least within the Thoracic Director's Association, of having a completely contained and intact electronic prerequisite and requisite monitored curriculum.

The technology is standard right now and it is really ripe to be used for life-long learning, both in terms of the ability to put readily available content on the Web and also in terms of our ability to track user's performance. The opportunity is there for any specialty or organization wishing to avail themselves of it.

Dr. Richard J. Finley (Vancouver, British Columbia, Canada): Dr. Gold, I want to congratulate you and the TSPD for showing us the future of surgical education.

I have 2 questions about the study. In your analysis, as no of the experimental group used a textbook alone for their educational process and only used the video or case studies in 5% of cases, I ask you: Why?

And the second question is: Are we not seeing a bias in the experimental group that has updated information in the electronic textbook versus the control group, who may have been using an older textbook?

Dr. Jeffrey P. Gold (Bronx, New York): This is a very small study group. Less than half of the 69 residents who were randomized to use the curriculum used it more than 20 times, which would be once every second week. There are many possible explanations for this. And indeed utilization over the 3 years since this initial group was randomized has gone up dramatically. Only in those residents who used it did we see the trend of improved performance on the examinations and the other benefits that the program directors were able to assess.

Yes, there are many differences between content that is available on the Web and in a standard textbook. That is one of the major advantages of Web based education, in that one can, in an hour, remove or add a segment of education and keep your content as current as possible.

Dr. Henry L. Laws (Birmingham, Alabama): Dr. Gold, I would like to congratulate you and your colleagues on undertaking this task and working through it. I think it is tremendous. Secondly, I would like to thank you for being sensitive to the needs of the general surgical educators, as you were, while considering these major changes in the thoracic surgical curriculum. We appreciate that.

Apparently the prospective thoracic residents spent an average of 54 hours studying the material provided them. That is an easy week or a hard week, however you want to look at it. A substantial number spent no time. I do think if people have formal, instructor-led training, as much as I hate to admit it, they tend to be more uniformly involved in the material. Would you do better to have everyone attend a 2-weeks course before beginning a thoracic residency?

Will this or is this material—and I gather it is not—available to other residents who just want to look at it in earlier years as they progress through their surgical training?

I think this is a great presentation, but mainly it is great work on the part of you and your colleagues.

Dr. Jeffrey P. Gold (Bronx, New York): Yes, I think we would like to seclude these residents into a nice retreat setting and get their attention for 2 weeks and teach them what we believe we need to teach them. But the practicality of doing that is very difficult. Also, it doesn't give them a permanent tool that they can take away with them and refer back to from time to time. So for as much a practical reason as anything else we chose to administer this using distance-learning or e-Learning.

Fortunately, the answer to your second question is that this information is available to all who wish to use it on our Web site. The security CD ROM system is only necessary if the resident wants to track their use of the curriculum. But it is available to all surgery residents, to all thoracic surgery residents, to anybody that wishes to use it. The way the determination of what was prerequisite and what was core or requisite was made is that we tried very consciously to put content into the prerequisite curriculum relevant to the remainder of the general surgery residency, to try to have as much programmatic overlap as possible.

Dr. Charles E. Lucas (Detroit, Michigan): Thirty-five years ago, Alec Wall, who believed the most important function of any chairman is the education of students and residents, spearheaded the development of a core curriculum. I was assigned the trauma part. His enthusiasm was quite remarkable, so that each of us looked upon the core curriculum as our spouse on a honeymoon. But, you know, honeymoons come to an end and the correct answers change every year, so that within 10 years the curriculum was a bit obsolete. So my one question is: What system have you put in place for the maintenance of the system so that it doesn't become obsolete?

Dr. Jeffrey P. Gold (Bronx, New York): We at the Thoracic Director's Association are very concerned about maintaining contemporary material in all of our curriculum projects. As such, the committees that structured and implemented the curriculum have been converted to editorial boards. It is their job, at periodic intervals, which has now been determined at least for the first go-around to be first 3 years, to completely revise the curriculum. So as we speak the Prerequisite Curriculum Editorial Board is tearing apart these original 75 segments, hopefully trashing a number of them, and actively at work developing some replacement sections to be sure that we have a current and up-to-date curriculum.