Simulators: a New Use for an Old Paradigm

Resident surgical skills’ training is about to undergo a fundamental change from what has been practiced before. It has been determined that surgical education is going to shift from a time-based training paradigm (5-year general surgery residency program) to a criterion-based model with an emphasis on obtaining and maintaining competency. The structure of the new paradigm will feature several years of a basic surgical curriculum followed by specialty training (general surgery, plastic surgery, and other specialties). The final form of the curriculum, though, has yet to be decided.

The cause for a shift in surgical training is multifactorial in character and has its roots in events of the 1980s and 1990s. The “Libby Zion” case initially brought the matter of residency training to public consciousness. In 1984, Ms. Zion presented to the New York Hospital with what appeared to be minor complaints of fever and earache. Eight hours later, she was dead.

There were allegations that the house staff responsible for Ms. Zion's care were overworked and undersupervised. A grand jury was convened to investigate these allegations. The grand jury refused to indict the doctors on criminal charges and their licenses were not revoked. However, the grand jury did indict the system of residency training, and the case became a rallying cry for graduate medical education reform. A special commission, headed by Bertrand N. Bell, a professor of medicine at Albert Einstein College of Medicine, was created and issued a series of recommendations on resident supervision and work hours.

New York state authorities heeded the special commission, and the New York Health Code of 1989 included regulations restricting resident work hours to 80 hours per week. In addition, the code specified that residents were to have one day a week free of clinical responsibilities and placed limits on the number of night calls in house.¹

Additional powerful forces over the past decade have combined to further stimulate a reexamination of the way surgical residents are taught.² These forces include a wellarticulated concern by many (governmental agencies, insurers, and the public) for patient safety and an intolerance to error. Also the “medical malpractice crisis” (fueled by increased complexity of cases, constantly changing technology), and the realities of managed care (cost constraints) have contributed to a reevaluation of how surgical training is accomplished.

In searching for answers, many have pointed out the positive experience of the airline industry with simulators and pilot training. Airplanes have grown safer, aircraft engines have become more reliable, and new procedures dealing with the prevention of errors have been adopted. All have contributed to airline passenger safety. However, one of the most important factors in pilot training–which has some similarities with surgical resident training– has been the use of sophisticated simulators to train pilots without those pilots having to actually fly an airplane. Many surgical educators now believe that this technology can be adapted to help train surgical residents.

But it will take more than adopting a particular training aid to accomplish an improvement in surgical training. Much more.

The Accreditation Council for Graduate Medical Education (ACGME) and various specialty Residency Review Committees (RRC) have begun the process of shifting from a time-based residency program to that of a competency-based paradigm. In 2002, the 6-core competencies of Patient care, Medical knowledge, Professionalism, Interpersonal and communication skills, Practice-based learning, and Systems-based practice were introduced into all residency training programs in the United States. These competencies have been inculcated into resident learning since 2002 and will continue for all future trainees. The groundwork for a competency-based education has been laid.

To accomplish the shift to a competency-based training mode, however, an entirely new paradigm for teaching surgical skills will have to be developed. The outlines of that development are now beginning to take shape and involve recognition that competency lies at the heart of surgical training and that surgical skills should be developed early beginning before a surgical resident enters an operating room.

To address these issues, the American College of Surgeons (ACS) and American Board of Surgery (ABS) have embraced the concept of simulation and the use of simulators to train and assess surgical skills. Along with other interested groups, such as the Association of Program Directors in Surgery (APDS), an ad hoc Committee on Simulator Assessment has begun development of a standardized curriculum to integrate simulators in surgical training. An ad hoc Committee on Simulator Centers will be responsible to review and certify education centers as either a Level 1 (comprehensive center) or a Level 2 center (training laboratory).

Importantly Competency-based education (CBE) is a concept that uses an approach to instruction and assessment that places primary emphasis on identifying and measuring specific learning outcomes or competence. The ACGME has identified 5 characteristics that are descriptive of teaching from a perspective of CBE. In CBE, teaching and learning are:

1. Explicit and clearly aligned with expected competencies;

2. Criteria-driven, focusing on accountability in reaching benchmarks and, ultimately, competence;

3. Grounded in “real-life” experiences;

4. Focused on fostering the learners’ ability to self-assess;

5. Individualized, providing more opportunities for independent study.³

Satava has suggested that a standardized simulator curriculum should include:

1. Metrics specific to the skill being taught;

2. Errors common for the skill set;

3. Specific curriculum for training the surgical skill;

4. A method to capture outcomes;

5. Validation methodology.⁴

The adoption of any new paradigm, unfortunately, usually implies learning a new language to communicate in that paradigm. The same is true for simulators, and it is important to assure that all interested parties that use simulators are able to communicate with one another utilizing the same language.⁵

For example, metrics refers to how we measure the performance of a simulator, which can then be used for formative (helping to shape, develop, or mold) or summative (final) assessment purposes. Metrics should be assessed for reliability and validity. Reliability reflects consistency of results when repeat examinations are performed on the same subject under the same conditions. Causes of inconsistency can include intraobserver variation (test-retest reliability), interobserver variation (interrater reliability), instrument and set-up variations, and inherent variation.

Simulator metrics (test scores for example) are used to provide an estimate of performance in the real world. Validity of simulators refers to the “extent to which a measurement, test, or study measures what it purports to measure.” In addition, there are several further refinements of validity that are necessary to understand when examining the effects of simulation.

Construct validity: The degree to which an instrument measures the characteristic being investigated; the extent to which the conceptual definition matches the operational definition. This is usually measured by comparing performance scores between groups expected to differ in skill, eg, junior versus senior residents.

Content validity: Verification that the method of measurement actually measures what it is expected to measure, covering all areas under investigation reasonably and thoroughly. Does the simulator (test) evaluate the appropriate (specific) content and breadth of content?

Face validity: A type of content validity, determining the suitability of a given instrument as a source of data on the subject under investigation, using common-sense criteria. On the face of it, do the metrics seem credible measures of the construct in question.

External validity: The extent to which study results can be generalized beyond the sample used in the study.

Internal validity: The extent to which the effects detected in a study are truly caused by the treatment or exposure in the study sample, rather than being due to other biasing effects of extraneous variables.

Concurrent validity: How well do test scores reflect performance as measured by another accepted or widely used measure made at the same time. Comparison of scores on another simulator, an animate model, or OSCE.

Predictive validity: How well do test scores predict those properties that they are designed to measure (technical skills in the OR)?⁶

Most educators agree that an ideal surgical simulator is one that outlines an exercise that reproduces, under laboratory conditions, a surgical procedure. It represents a simplified reality that does not involve direct patient care or contact.

Freid has suggested that when assessing a simulator for purchase or incorporation into a residency curriculum, the following questions need be addressed:

1. What are the specific educational objectives?

2. How well does the simulator meet the educational objectives?

3. How can student performance be tracked in the operating room and what measures can be used to follow performance over time?

4. Does the simulator provide feedback (metrics) to predict that performance improvement on the simulator will correlate with improved performance of the techniques taught?⁵

Not all simulators are alike, nor do they need to be complicated. Knot-tying practice on a chair has served as a simulator for generations of surgical residents, and has served them well. Simulators can be valuable tools in residency training. But to go a step beyond simple simulation, it is necessary to have valid information that the simulation exercise is valuable in teaching the desired skill set or technique. Validation data provide the essential information that enables identification of an appropriate simulator, measures performance, and easily fits into a curriculum for residency training programs.

Surgical simulators will become an important part of residency training. They have the potential to enhance the quality of education in surgical anatomy and the teaching of basic and advanced open and laparoscopic technical skills, as well as to serve as preoperative planning tools for complex procedures. All will serve to improve patient safety.

Finally, the use of simulators could provide an objective certification/recertification tool to determine the surgical skills of established practitioners, similar to those of the airline industry.⁷ The science of simulators is in its infancy, but the possibility of simulators serving as surrogate patients is real and within reach.