Development of a New Valid, Reliable, and Internationally Applicable Assessment Tool of Residents’ Competence in Ophthalmic Surgery (An American Ophthalmological Society Thesis)

Abstract

Purpose:

To test the validity and reliability of a new tool for assessing residents’ competence in ophthalmic surgery. Changing paradigms of ophthalmic education in the United States have influenced worldwide ophthalmic education and necessitated new methods of assessing resident competence. Accordingly, a new tool for assessing residents’ competence in ophthalmic surgery (phacoemulsification) that could be applicable internationally was developed. We hypothesize that this instrument is valid and reliable.

Methods:

A panel of six international content experts adapted a previously published tool for assessing phacoemulsification. The tool (called the International Council of Ophthalmology’s Ophthalmology Surgical Competency Assessment Rubric, or ICO-OSCAR:phaco) was reviewed by 12 international content experts for their constructive comments, which were incorporated to ensure content validity. Ten expert cataract surgery teachers then graded six recorded phacoemulsification surgeries with the ICO-OSCAR:phaco to investigate inter-rater reliability.

Results:

The coefficient alpha statistic (a measure of reliability/internal consistency) for the ICO-OSCAR:phaco as a whole was 0.92, and 17 of its 20 dimensions had alpha coefficients greater than 0.70.

Conclusions:

The ICO-OSCAR:phaco is a valid and reliable assessment tool that could be applied internationally to satisfy the global need of new instruments to comply with emerging trends in ophthalmic education. A toolbox of similar surgical competency assessment tools is being developed.

INTRODUCTION

Several factors are creating change in global ophthalmic medical education, and some of these factors have originated in changes occurring in the United States. In 1999, the US Accreditation Council for Graduate Medical Education (ACGME) announced the Outcomes Project. Previously, accreditation was based on demonstrating the potential of a residency program to educate, the existence of adequate resources, curriculum, and a structured learning experience. The Outcomes Project refocused accreditation on whether residents were actually learning by looking at outcomes of their education. An initial component of this project was the identification of six general competencies deemed essential to medical education: (1) medical knowledge, (2) patient care, (3) practice-based learning, (4) interpersonal and communication skills, (5) professionalism, and (6) systems-based practice.¹ Residency programs in the United States are expected to teach these competencies and develop tools to assess competence in these areas.

Global dissemination and acceptance of these ideas and assessment tools have been greatly facilitated by the rapid communication and educational opportunities possible via the Internet and by the efforts of the International Council of Ophthalmology (ICO).

Because of these factors and to address international educational demand, we developed a new tool designed to teach and assess resident skill in phacoemulsification, the ICO-Ophthalmology Surgical Competency Assessment Rubric (ICO-OSCAR:phaco) (Table 1).² We have shown that the ICO-OSCAR:phaco is valid. We hypothesize that this tool can be used globally with good inter-rater reliability.

TABLE 1.

ICO–OPHTHALMOLOGY SURGICAL COMPETENCY ASSESSMENT RUBRIC:PHACOEMULSIFICATION (ICO-OSCAR:PHACO)

DATE___________________ RESIDENT___________________ EVALUATOR___________________		NOVICE (SCORE = 2)	BEGINNER (SCORE = 3)	ADVANCED BEGINNER (SCORE = 4)	COMPETENT (SCORE = 5)	NOT APPLICABLE. DONE BY PRECEPTOR (SCORE= 0)
1	Draping	Unable to start draping without help.	Drapes with minimal verbal instruction. Incomplete lash coverage.	Lashes mostly covered, drape at most minimally obstructing view.	Lashes completely covered and clear of incision site, drape not obstructing view.
2	Incision & Paracentesis: Formation & Technique	Inappropriate incision architecture, location, and size.	Leakage and/or iris prolapse with local pressure, provides poor surgical access to and visibility of capsule and bag.	Incision either well placed or nonleaking but not both.	Incision parallel to iris, self-sealing, adequate size, provides good access for surgical maneuvering.
3	Viscoelastic: Appropriate Use and Safe Insertion	Unsure of when, what type, and how much viscoelastic to use. Has difficulty accessing anterior chamber through paracentesis.	Requires minimal instruction. Knows when to use but administers incorrect amount or type.	Requires no instruction. Uses at appropriate time. Administers adequate amount and type. Cannula tip in good position. Unsure of correct viscoelastic if multiple types available.	Viscoelastics are administered in appropriate amount and at the appropriate time with cannula tip clear of lens capsule and endothelium. Appropriate viscoelastic is used if multiple types of viscoelastics are available.
4	Capsulorrhexis: Commencement of Flap & Follow-through.	Instruction required, tentative, chases rather than controls rhexis, cortex disruption may occur.	Minimal instruction, predominantly in control with occasional loss of control of rhexis, cortex disruption may occur.	In control, few awkward or repositioning movements, no cortex disruption.	Delicate approach and confident control of the rhexis, no cortex disruption.
5	Capsulorrhexis: Formation and Circular Completion	Size and position are inadequate for nucleus density & type of implant, tear may occur.	Size and position are barely adequate for nucleus density and implant type, difficulty achieving circular rhexis, tear may occur.	Size and position are almost exact for nucleus density and implant type, shows control, requires only minimal instruction.	Adequate size and position for nucleus density & type of implant, no tears, rapid, unaided control of radialization, maintains control of the flap and AC depth throughout the capsulorrhexis.
6	Hydrodissection: Visible Fluid Wave and Free Nuclear Rotation	Hydrodissection fluid not injected in quantity nor place to achieve nucleus rotation.	Multiple attempts required, able to rotate nucleus somewhat but not completely. Tries to manually force rotation before adequate hydrodissection.	Fluid injected in appropriate location, able to rotate nucleus but encounters more than minimal resistance.	Ideally see free fluid wave but adequate if free nuclear rotation with minimal resistance is achieved. Aware of contraindications to hydrodissection.
7	Phacoemulsification Probe and Second Instrument: Insertion Into Eye	Has great difficulty inserting the probe or second instrument, AC collapses, may damage wound, capsule, or Descemet’s membrane.	Inserts the probe or second instrument after some failed attempts, may damage wound, capsule, or Descemet’s membrane.	Inserts probe and second instrument on first attempt with mild difficulty, no damage to wound, capsule, or Descemet’s membrane.	Smoothly inserts instruments into the eye without damaging the wound or Descemet’s membrane.
8	Phacoemulsification Probe and Second Instrument: Effective Use and Stability	Tip frequently not visible, has much difficulty keeping the eye in primary position, and uses excessive force to do so.	Tip often not visible, often requires manipulation to keep eye in primary position.	Maintains visibility of tip at most times, eye is generally kept in primary position with mild depression or pulling on the globe.	Maintains visibility of instrument tips at all times, keeps the eye in primary position without depressing or pulling up the globe.
9	Nucleus: Sculpting or Primary Chop	Frequently incorrect power used during sculpting, applies power at inappropriate times, excessive phaco probe movement causes constant eye/nucleus movement, unable to engage nucleus (chop method) or the groove is of inadequate depth or width (divide and conquer), cannot control phacodynamics. Unable to correctly work foot pedals.	Moderate error in power used while sculpting, tentative, frequent eye/nucleus movement produced by phaco tip, difficult to engage nucleus (chop technique) or groove adequately after many attempts (divide and conquer), poor control of phacodynamics with frequent anterior chamber depth fluctuations. Has difficulty working foot pedals.	Uses correct power with minimal error when sculpting, occasional eye/nucleus movement caused by phaco tip, some difficulty in engaging or holding nucleus (chop method) or groove adequate with minimal repeat attempts, fairly good control of phacodynamics with occasional anterior chamber depth change. Minimal mistakes using foot pedals.	Sculpting is performed using adequate ultrasound power regulated by the pedal, with forward movements that do not change the eye position or push the nucleus, the nucleus is safely engaged (with chop method) or the groove is appropriate in depth and width (divide and conquer technique), phacodynamics are controlled as evidenced by the internal anterior chamber environment. Adept at foot pedal control.
10	Nucleus: Rotation and Manipulation	Unable to rotate nucleus.	Able to rotate nucleus partially and with zonular stress.	Able to rotate nucleus fully but with zonular stress.	Nucleus is safely and efficiently manipulated producing minimal stress on zonules and globe.
11	Nucleus: Cracking or Chopping With Safe Phacoemulsification of Segments	CRACKING: Grooves are not centered or deep enough and go into epinucleus, nucleus is constantly displaced from central position, unable to crack nucleus at all, eye constantly moving. CHOPPING: Always endangers or engages adjacent tissue, unable to accomplish chop of any piece. SEGMENT PHACOEMULSIFICATION: Produces significant wound burn, great difficulty pursuing fragments around the anterior chamber and into the bag, poor awareness of second instrument tip and difficulty keeping the second hand instrument under the phaco tip.	CRACKING: Some grooves are centered and deep enough and some go into epinucleus, displaces nucleus in most grooves, attempts to split nucleus with instruments too shallow in groove, able to crack portion of nucleus, eye often moving. CHOPPING: Endangers or engages adjacent tissue in most chops, able to accomplish chop of some pieces. SEGMENT PHACOEMULSIFICATION: Produces light wound burn, pursues most fragments around the AC and into the bag, the second hand instrument is sometimes under the phaco tip.	CRACKING: Most grooves are centered and deep enough, rarely goes into epinucleus, rarely displaces nucleus, sometimes attempts to split in mid-nucleus but succeeds, eye usually in primary position. CHOPPING: Endangers or engages adjacent tissue in some chops, able to accomplish chop of most pieces. SEGMENT PHACOEMULSIFICATION: Produces minimal wound burn, pursues some fragments around the AC and into the bag, the second hand instrument is usually under the phaco tip.	CRACKING: Grooves are centered, deep enough to ensure cracking, length does not reach epinucleus, nucleus is not displaced from central position, places instruments deep enough to easily and successfully crack nucleus, eye stays in primary position. CHOPPING: Nucleus engaged and vertical or horizontal chop technique undertaken with no inadvertent engagement of adjacent tissue (especially capsule). Full-thickness nuclear chop of all pieces in a controlled and fluid manner. SEGMENT PHACOEMULSIFICATION: No wound burns, pieces are “floated” to the tip without “pursuing” the fragments around the anterior chamber and the bag, the second hand instrument is kept under the phaco tip to prevent posterior capsule contact if surge arises.
12	Irrigation and Aspiration Technique With Adequate Removal of Cortex	Great difficulty introducing the aspiration tip under the capsulorrhexis border, aspiration hole position not controlled, cannot regulate aspiration flow as needed, cannot peel cortical material adequately, engages capsule or iris with aspiration port.	Moderate difficulty introducing aspiration tip under capsulorrhexis and maintaining hole up position, attempts to aspirate without occluding tip, shows poor comprehension of aspiration dynamics, cortical peeling is not well controlled, jerky and slow, capsule potentially compromised, prolonged attempts result in minimal residual cortical material.	Minimal difficulty introducing the aspiration tip under the capsulorrhexis, aspiration hole usually up, cortex will engage for 360 degrees, cortical peeling slow, few technical errors, minimal residual cortical material.	Aspiration tip is introduced under the free border of the capsulorrhexis in irrigation mode with the aspiration hole up, aspiration is activated in just enough flow as to occlude the tip, efficiently removes all cortex, the cortical material is peeled gently toward the center of the pupil, tangentially in cases of zonular weakness.
13	Lens Insertion, Rotation, and Final Position of Intraocular Lens	Unable to insert IOL, unable to produce adequate incision for implant type. NONFOLDABLE: Unable to place the lower haptic in the capsular bag, unable to rotate the upper haptic into place. FOLDABLE: Unable to load IOL into injector or forcep, no control of lens injection, doesn’t control tip placement, lens is not in the capsular bag or is injected upside down.	Insertion and manipulation of IOL is difficult, eye handled roughly, anterior chamber not stable, repeated attempts result in borderline incision for implant type. NONFOLDABLE: Repeated hesitant attempts result in lower haptic in the capsular bag, upper haptic is rotated into place but with excessive force on capsulorrhexis and zonules and repeated attempts are necessary. FOLDABLE: Difficulty loading IOL into injector or forcep, hesitant, poor control of lens injection, difficulty controlling tip placement, excessive manipulation required to get both haptics into capsular bag.	Insertion and manipulation of IOL is accomplished with minimal anterior chamber instability, incision just adequate for implant type. NONFOLDABLE: The lower haptic is placed inside the capsular bag with some difficulty, upper haptic is rotated into place with some stress on the capsulorrhexis and zonule fibers. FOLDABLE: Minimal difficulty loading IOL into injector of forcep, hesitant but good control of lens injection, minimal difficulty controlling tip placement, both haptics are in the capsular bag.	Insertion and manipulation of IOL is performed in a deep and stable anterior chamber and capsular bag, with incision appropriate for implant type. NONFOLDABLE: The lower haptic is smoothly placed inside the capsular bag, upper haptic is rotated into place without exerting excessive stress to the capsulorrhexis or the zonule fibers. FOLDABLE: Able to load IOL into injector or forcep, lens is injected in a controlled fashion, fixation of IOL is symmetric, the optic and both haptics are inside the capsular bag.
14	Wound Closure (Including Suturing, Hydration, and Checking Security as Required)	If suturing is needed, instruction is required and stitches are placed in an awkward, slow fashion with much difficulty, astigmatism, bent needles, incomplete suture rotation and wound leakage may result, unable to remove viscoelastics thoroughly. Unable to make incision watertight or does not check wound for seal. Improper final IOP.	If suturing is needed, stitches are placed with some difficulty, resuturing may be needed, questionable wound closure with probable astigmatism, instruction may be needed, questionable whether all viscoelastics are thoroughly removed. Extra maneuvers are required to make the incision watertight at the end of the surgery. May have improper IOP.	If suturing is needed, stitches are placed with minimal difficulty tight enough to maintain the wound closed, may have slight astigmatism, viscoelastics are adequately removed after this step with some difficulty. The incision is checked and is watertight or needs minimal adjustment at the end of the surgery. May have improper IOP.	If suturing is needed, stitches are placed tight enough to maintain the wound closed, but not too tight as to induce astigmatism, viscoelastics are thoroughly removed after this step. The incision is checked and is watertight at the end of the surgery. Proper final IOP.
15	Global Indices Wound Neutrality and Minimizing Eye Rolling and Corneal Distortion	Nearly constant eye movement and corneal distortion.	Eye often not in primary position, frequent distortion folds.	Eye usually in primary position, mild corneal distortion folds occur.	The eye is kept in primary position during the surgery. No distortion folds are produced. The length and location of incisions prevents distortion of the cornea.
16	Eye Positioned Centrally Within Microscope View	Constantly requires repositioning.	Occasional repositioning required.	Mild fluctuation in pupil position.	The pupil is kept centered during the surgery.
17	Conjunctival and Corneal Tissue Handling	Tissue handling is rough and damage occurs.	Tissue handling borderline, minimal damage occurs.	Tissue handling decent but potential for damage exists.	Tissue is not damaged nor at risk by handling.
18	Intraocular Spatial Awareness	Instruments often in contact with capsule, iris, and corneal endothelium, blunt second hand instrument not kept in appropriate position.	Occasional accidental contact with capsule, iris, and corneal endothelium, sometimes has blunt second hand instrument between the posterior capsule and the activated phaco tip.	Rare accidental contact with capsule, iris, and corneal endothelium. Often has blunt second hand instrument between the posterior capsule and the activated phaco tip.	No accidental contact with capsule, iris, and corneal endothelium, when appropriate, a blunt, second hand instrument is always kept between the posterior capsule and the tip of the phaco when the phaco is activated.
19	Iris Protection	Iris constantly at risk, handled roughly.	Iris occasionally at risk. Needs help in deciding when and how to use hooks, ring, or other methods of iris protection.	Iris generally well protected. Slight difficulty with iris hooks, ring, or other methods of iris protection.	Iris is uninjured. Iris hooks, ring, or other methods are used as needed to protect the iris.
20	Overall Speed and Fluidity of Procedure	Hesitant, frequent starts and stops, not at all fluid.	Occasional starts and stops, inefficient and unnecessary manipulations common, case duration about 60 minutes.	Occasional inefficient and/or unnecessary manipulations occur, case duration about 45 minutes.	Inefficient and/or unnecessary manipulations are avoided, case duration is appropriate for case difficulty. In general, 30 minutes should be adequate.
Comments:________________________________________________________________

METHODS

This study was deemed exempt from institutional review board (IRB) review by the University of Cincinnati IRB. The study adheres to the Declaration of Helsinki and all federal or state laws in our countries.

DEVELOPMENT OF THE ASSESSMENT TOOL

Saleh and colleagues3 described an assessment tool called the Objective Structured Assessment of Cataract Surgical Skill (OSACSS). This tool breaks down the phacoemulsification procedure into 20 steps that are scored on a 5-point Likert scale. The scale anchors are: 1 = poorly or inadequately performed, 3 = performed with some errors or hesitation, and 5 = performed well with no prompting or hesitation. While clearly a good start, the scale is simple and not well defined (there are no scale anchors for scores of 2 or 4). Also, there is no behavioral or skill-based rubric for the evaluators to use when assessing the resident’s competence.

Starting with the OSACSS 20-step skill breakdown, we have developed and validated an instrument for assessing ophthalmology residents’ competency in performing cataract surgery.² We modified the Dreyfus scale to reflect levels of resident skill as novice, beginner, advanced beginner, and competent (we felt that no resident becomes expert during the course of their training), creating a comprehensive, globally applicable rubric, with the addition of behavioral anchors for each level in each step of the surgical procedure.

DETERMINATION OF CONTENT VALIDITY

A group of content experts, representing the United States, England, Argentina, and the United Arab Emirates, worked together to develop the scale anchors and behaviorally based rubric for the OSACSS. To facilitate collaboration, the drafts of the rubric were developed online via Google communication sites. Once drafted, content and face validity were achieved by having an international panel of 15 experts review the draft instrument and provide feedback.² The international panel was composed of representatives from North America, Latin America, Asia, the Middle East, Europe, and Africa. After incorporating suggestions from the international panel, a final document, the ICO–Ophthalmology Surgical Competency Assessment Rubric–phacoemulsification (ICO-OSCAR:phaco) was produced (Table 1).

DETERMINATION OF RELIABILITY

A panel of 10 international experts was selected to achieve a broad geographic representation. Countries represented were Argentina (2), United States (2), United Kingdom (1), United Arab Emirates (1), India (1), Egypt (1), China (1), and Australia (1). Experts were experienced cataract surgeons. Each expert received Internet links to six different videotaped resident phacoemulsification surgeries and the ICO-OSCAR:phaco assessment tool. The phacoemulsification procedures were selected from residents at different levels of experience to try to represent a range of surgical skill.

To assess the inter-reliability of the ICO-OSCAR:phaco, we calculated coefficient Cronbach alpha statistics.⁴ This statistic, which has become the measure of choice for assessing the reliability of a multi-item scale, provides an indication of the degree to which a scale of survey or test items reliably measures an underlying factor or construct.⁵ The magnitude of the statistic is based on the average correlation among all of the items in the scale (internal consistency) as well as the number of items included in the scale. A value of 0.70 is acceptable for newly developed scales.⁶ To address the issue of the small amount of missing data, we have imputed the missing data using a procedure called “mice” (multiple imputations by chained equations) available in the computing software R, which is freely available. The complete data sets are generated randomly by simulating certain conditional distributions. Cronbach alpha was calculated for each completed data set. All calculations were done using the computing software R.

RESULTS

DETERMINATION OF VALIDITY

The international panel’s comments on the initial tool draft included both general and specific suggestions. Several reviewers suggested adding new categories to the 20 published in the OSACSS, and two reviewers felt the tool might be too extensive and burdensome to complete. However, the consensus of the experts was to not add categories. The tool is fairly extensive, but we felt it has to be to achieve our goal of teaching and assessing with the same tool. There were numerous specific suggestions regarding many of the rubric’s behavioral descriptors. All expert comments were considered, and the authors incorporated appropriate suggestions, thus establishing a level of face and content validity.²

DETERMINATION OF INTER-RATER RELIABILITY

Sixty ICO-OSCAR:phaco forms were completed. Fifty-four data points out of a possible 1200 (4.5%) were missing. The calculated raw coefficient alpha statistic for the ICO-OSCAR:phaco as a whole was 0.92. Individual dimension (each surgical procedure step) alpha statistics are shown in Table 2. Seventeen of 20 alpha statistics were greater than 0.70. Three of the dimensions had alphas of less than 0.70: step 4 (Capsulorrhexis: Commencement of Flap & Follow-through) = 0.62, step 5 (Capsulorrhexis: Formation and Circular Completion) = 0.64, and step 17 (Conjunctival and Corneal Tissue Handling) = 0.47.

TABLE 2.

INTERNATIONAL COUNCIL OF OPHTHALMOLOGY’S OPHTHALMOLOGY SURGICAL COMPETENCY ASSESSMENT RUBRIC: PHACOEMULSIFICATION CRONBACH ALPHAS BY ITEM

	ITEM	CRONBACH ALPHA
1	Draping	0.91
2	Incision & Paracentesis: Formation & Technique	0.84
3	Viscoelastic: Appropriate Use and Safe Insertion	0.76
4	Capsulorrhexis: Commencement of Flap & Follow-through.	0.62
5	Capsulorrhexis: Formation and Circular Completion	0.64
6	Hydrodissection: Visible Fluid Wave and Free Nuclear Rotation	0.94
7	Phacoemulsification Probe and Second Instrument: Insertion Into Eye	0.79
8	Phacoemulsification Probe and Second Instrument: Effective Use and Stability	0.88
9	Nucleus: Sculpting or Primary Chop	0.96
10	Nucleus: Rotation and Manipulation	0.90
11	Nucleus: Cracking or Chopping With Safe Phacoemulsification of Segments	0.97
12	Irrigation and Aspiration Technique With Adequate Removal of Cortex	0.98
13	Lens Insertion, Rotation, and Final Position of Intraocular Lens	0.87
14	Wound Closure (Including Suturing, Hydration, and Checking Security as Required)	0.88
	Global Indices
15	Wound Neutrality and Minimizing Eye Rolling and Corneal Distortion	0.93
16	Eye Positioned Centrally Within Microscope View	0.94
17	Conjunctival and Corneal Tissue Handling	0.47
18	Intraocular Spatial Awareness	0.86
19	Iris Protection	0.77
20	Overall Speed and Fluidity of Procedure	0.93

DISCUSSION

In the following paragraphs we provide a historical background of medical education in the United States to emphasize how we believe the ACGME’s Outcomes Project and competencies model have disseminated and influenced changes in ophthalmic education worldwide, leading to a global need for new valid and reliable tools to assess residents’ competence. This need led us to develop the ICO:OSCAR.

DEVELOPMENT OF MODERN OPHTHALMIC EDUCATION IN THE UNITED STATES

In the early 20th century, ophthalmic education in the United States was unstructured and of variable quality. Training programs were not associated with medical schools at that time, and specialists’ training often consisted of preceptorships or 6-week courses designed to convert the general practitioner to an ophthalmologist. In 1913, the American Academy of Ophthalmology and Otolaryngology (AAOO) recognized the poor state of ophthalmic education and formed the Education in Ophthalmology Committee,⁷ which was charged with upgrading postgraduate education.

The formation of the American Board for Ophthalmic Examinations and recommendations made by the American Medical Association led to an increase in the number and standardization of ophthalmic residencies. An AAOO Teachers Section was formed in 1930 aimed at improving the state of graduate education.⁸ In 1957, the American Board of Ophthalmology (ABO) standardized ophthalmic postgraduate training as 1 year of internship and 3 years of ophthalmology residency.⁹

In 1968, the first annual Ophthalmology Residency-in-Training Examination was sponsored by the Association of University Professors of Ophthalmology (AUPO). In 1969, the AAOO took over sponsorship of this examination, which was meant to be a means to determine knowledge gaps and to help guide remediation.¹⁰ A separate Practitioner assessment procedure, the Ophthalmic Knowledge Self-Assessment Program, was developed in 1970. Subsequently, these two assessments were merged in 1972 to become the Ophthalmic Knowledge Assessment Program (OKAP). The Basic and Clinical Science Course was devised to integrate basic science and clinical application and is still used by all US ophthalmology residency programs.⁷ By the end of the 1990s, however, concerns arose about the ability of new graduates to meet the demands of today’s practice environment.

THE ACCREDITATION COUNCIL FOR GRADUATE MEDICAL EDUCATION OUTCOMES PROJECT

The ACGME is a national, private, not-for-profit organization charged with accrediting medical resident training in the United States. Composed of a Board and Residency Review Committees that set, monitor, and enforce standards for residencies in each specialty, its representatives visit and review each program every 2 to 5 years, depending on how well a program is doing.¹¹ In 1999, the ACGME and the American Board of Medical Specialties introduced a general competency and outcome assessment initiative known as the Outcome Project,¹ in an effort to enhance residency education and accreditation effectiveness by increasing emphasis on educational outcomes as opposed to process. The Project was also a response to concerns aforementioned about new graduates’ ability to meet the demands of today’s practice environment. The driving force for the development of the competencies arose from three main advocates for change: the market, the government, and the public.^12–15

A major component of this project was the identification of six core competencies of physician training: (1) patient care, (2) medical knowledge, (3) professionalism, (4) communication skills, (5) practice-based learning, and (6) systems-based practice. Residency programs are supposed to ensure competence in these domains by collecting performance data that reliably and accurately depicts the resident’s ability to care for patients and to work effectively in health care delivery systems. The components of each core competency are listed in Table 3. During this same time period and for similar reasons, the Royal College of Physicians and Surgeons in Canada developed an analogous initiative known as the CanMEDS Physician Competency Framework.¹⁶

TABLE 3.

THE ACCREDITATION COUNCIL FOR GRADUATE MEDICAL EDUCATION CORE COMPETENCIES

Patient care

Communicate effectively, demonstrate caring and respectful behavior

Gather essential and accurate information

Make informed decisions about diagnostic and therapeutic interventions

Develop and carry out patient management plans

Perform competently medical and invasive procedures

Provide patient counseling and education

Use technology

Provide preventive and health maintenance services; work with other care providers to provide patient-focused care

Medical knowledge

Obtain biomedical, clinical, social-behavioral, and epidemiological knowledge; demonstrate investigatory and analytic thinking

Practice-based learning and improvement

Identify strengths, deficiencies, and limits in one’s knowledge and experience

Set learning and improvement goals

Identify and perform appropriate learning activities

Incorporate formative evaluative feedback into daily practice

Systematically analyze practice and implement changes to improve practice

Appraise and use scientific evidence

Use technology to optimize learning; participate in the education of patients, families, and other health professionals

Interpersonal and communication skills

Communicate effectively with patients, families, and the public

Communicate effectively with physicians, other health professionals, and health-related agencies

Work with other care providers as a team leader or member

Act in a consultative role to other physicians, health-related agencies, and policy makers; maintain medical records

Professionalism

Demonstrate respect, compassion, and integrity

Demonstrate responsiveness to patient needs that supersedes self-interest

Demonstrate accountability to patients, society, and the profession

Demonstrate excellence and ongoing professional development

Demonstrate adherence to ethical principles

Demonstrate sensitivity and responsiveness to diverse patient population

Demonstrate respect for patient privacy and autonomy

Systems-based practice

Coordinate patient care

Incorporate cost awareness and risk-benefit analysis

Advocate for quality patient care and optimal health care or public health systems

Work in interprofessional teams to enhance quality and safety

Participate in identifying system errors

Thus, the ACGME’s Outcome Project represented the first major change in postgraduate medical education since the early 1900s, when residency training became mandatory. Having excellent training resources and adequate numbers of patients was no longer enough; now the residency program and program director had to show that the resident is competent in the six core competencies by completion of the residency.

OPHTHALMOLOGY’S RESPONSE

Ophthalmology’s response to the ACGME mandate was fairly rapid and impressive. The AAO, ABO, AUPO, and individual ophthalmic educators combined to address the mandate and improve ophthalmology resident education and assessment in the United States. Some of the initiatives developed accordingly are the following.

Educating the Educators

Traditionally, medical educators learned how to teach by direct observation (role modeling) of their instructors. In general, there was no formal instruction on adult learning theory, teaching skills, or methods of assessment. In 2001, an innovative course, “Teaching and Learning in Ophthalmology,” was held during the AAO annual meeting in an attempt to enhance the participants’ ability to educate. Subsequently, an AAO annual meeting 90-minute symposium, “Teaching and Learning in Ophthalmology,” was sponsored by the AUPO. First held in 2003, this symposium has occurred annually and has addressed a variety of educational topics, including teaching and assessing the ACGME core competencies.

In 2003, Dr Andrew Lee organized the first Educating the Educators (EE) meeting, held in Iowa City. The goal of the meeting was to provide ophthalmology program directors more formal instruction on how to teach and assess their residents; specifically, the ACGME competencies were addressed and plans to meet the ACGME mandate were discussed. The second EE meeting was held in 2004 in Cincinnati with the aim to provide an annual forum for ophthalmic educators. In 2005, the AUPO endorsed the EE meeting, and beginning in February 2006, the one-day meeting occurs the day immediately preceding the annual AUPO meeting at the same venue. This has allowed more program directors to participate and has ensured viability of the event.

Program Director Recognition

One effect of the ACGME Outcomes Project was to increase the program directors’ workload. Unfortunately, in 2000, even before the complete ramifications of the ACGME competencies were fully realized, the turnover rate of US ophthalmology program directors was 33% in that year alone. Furthermore, the average tenure of the program director was 2.7 years.¹⁷

The AUPO recognized the importance of the Program Director to the future of ophthalmology and acted to remedy the situation. A Program Directors Advisory Council (later changed to the Program Directors Council) was formed in 2002 to provide organization and resources for the ophthalmology program director. Additionally, the Straatsma Award was created by the AAO and AUPO to recognize excellence in resident education. The program director recipient delivers a lecture during the AAO Teaching and Learning in Ophthalmology Symposium and at the annual AUPO meeting.

It was hoped the increased recognition and resources available to program directors would decrease turnover and increase average tenure of this crucial educational position. Indeed, in December 2009, one of the authors (K.C.G.) surveyed ophthalmology program directors as to their tenure and received responses from 109 of 114 program directors. Average tenure of this group had increased to 6.9 years.

Teaching and Assessing the Competencies

Prior to the introduction of the ACGME competencies, the majority of resident assessment was done by the faculty’s global rating of the residents and through the OKAP examination. However, with the introduction of the Outcomes Project, it became apparent that new assessment tools had to be developed: traditional tools were not appropriate for residents to demonstrate competence.

Shortly after introduction of the ACGME Outcomes Project, the AUPO formed a task force of program directors to address ways to meet the mandate. Subsequently, the ABO formed a Competency Task Force under the leadership of Richard Mills, MD, charged with developing assessment tools to help fulfill the ACGME requirements and to avoid the need for each ophthalmology program to duplicate efforts. This led to development of a valid and reliable tool to assess resident competence in performing a complete patient examination: the Ophthalmic Clinical Evaluation Exercise (OCEX).^18,19

Subsequently, a variety of surgical competency assessment tools were developed by groups of individual ophthalmic medical educators. Cremers and associates^20,21 developed the Objective Assessment of Skills in Intraocular Surgery (OASIS), and the Global Rating Assessment of Skills in Intraocular Surgery (GRASIS). Feldman and Geist²² described the Subjective Phacoemulsification Skills Assessment as an evaluative instrument designed specifically for intraoperative assessment of resident phacoemulsification cataract extraction surgery. Rogers and associates²³ have shown that changes in their surgical curriculum have decreased resident complications during cataract surgery. In 2008, Golnik and associates²⁴ performed a survey of ophthalmology program directors in the United States to determine current compliance with the ACGME competencies project, barriers to implementation, and recommendations for future collaborative assistance and work. The majority of respondents reported that they intend to expand the use of tools such as observed resident–patient interactions, 360° evaluation, and portfolios.

INTERNATIONAL EFFORTS FOR ENHANCING OPHTHALMIC EDUCATION

The International Council of Ophthalmology

Under the leadership of Bruce Spivey, MD (ICO president), ophthalmic education has become a main emphasis of the ICO. The efforts of this international organization are aimed not just at providing venues and resources to improve ophthalmic knowledge and procedural skills, but to improve the educational abilities of the individuals responsible for training the next generation of ophthalmologists. In 1999, the ICO developed a strategic plan, which included the development of International Curricula of Ophthalmic Education.²⁵ These curricula are designed to cover the education of the entire eye care team, including the ophthalmologist, allied eye care personnel, and medical students, and include continuing medical education. One intent of these curricula is to promote a shift from the traditional apprenticeship training model, where content and format are highly variable, to a curriculum-based system in which goals, expectations, knowledge base, competencies, and technical training are carefully defined, and to recommend the ACGME competencies’ model and ideas as good practice for residents’ training.

In 2004, one of the authors (K.C.G.) was approached by the ICO to help organize an Ophthalmology Program Directors meeting aimed at “educating the educators” regarding adult learning principles, curriculum development, teaching skills, and assessment techniques. Endorsement by the AAO, AUPO, the Pan-American Association of Ophthalmology (PAAO), and other supranational organizations has lent credibility to these meetings. Thus far, 21 meetings have been held in Latin America, Asia, Africa, and Europe. Approximately 1000 ophthalmic educators have participated. Outcomes from these meetings have included formation of national program director groups, further self-supported educational meetings, and opportunities to report on individual program progress at World Ophthalmology Congresses.

An extension of these meetings has developed as the ICO’s Conferences for Educators. These one-day meetings occur prior to every supranational meeting (Asia-Pacific Academy of Ophthalmology [APAO], European Society of Ophthalmology [SOE], Middle East Africa Council of Ophthalmology [MEACO], PAAO) and provide a permanent venue for all ophthalmic educators to continually update their teaching and assessing skills.

Organized by the ICO, the first World Ophthalmic Education Colloquium (WOEC) has been part of the World Ophthalmology Congresses since 2008. It includes 18 symposia dedicated to ophthalmic assistant, medical student, resident, fellow, and program director education. This one-day effort brought the concept of education to the forefront and provided a venue for collaboration of ophthalmic educators from around the world.

Another major ICO educational initiative is the task force on Emerging Technologies for Teaching and Learning, charged to explore new and emerging technologies for ophthalmic education and providing technical guidance for online continuing medical education courses and other learning experiences. The ICO’s Center for Ophthalmic Educators online resource center was developed to facilitate collaboration between resident educators. The target audience is ophthalmic educators interested in resident, fellow, medical student, and allied health education.

Other International Efforts

Under the leadership of President Richard Abbott, the PAAO formed a Resident Education in Latin America Committee in 2008, which was charged with improving residents’ education in Latin America. Activities thus far have included establishing a comprehensive list of residency programs in Latin America and surveying program directors as to their most immediate resident educational needs. The committee has completed a document, “PAAO Recommended Program Requirements for Graduate Medical Education in Ophthalmology,” which recommends programmatic and educational requirements. These include utilizing methods for teaching and assessing the ACGME’s 6 Competencies. Additionally, the PAAO has introduced symposia aimed at improving educational effectiveness at their biannual meeting.

In 2009 and 2010, respectively, MEACO and APAO also introduced residency program director meetings and ophthalmic education symposia into their congresses.

The European Board of Ophthalmology (EBO) has specifically been given the task of overseeing the standard of ophthalmology training in Europe. The EBO Strategic Plan for 2009–2010 contained several educational facets, including (1) harmonization of the level of education in all of its territory (create minimum curricular standards throughout Europe), (2) collaboration with societies such as the ICO and AAO for content production, (3) increasing the profile and awareness of education, and (4) providing “training the trainers” courses to increase educational abilities of resident educators.²⁶

INTERNATIONALLY APPLICABLE TEACHING AND ASSESSING TOOLS

After interacting with hundreds of program directors from around the world (during the authors’ coordination of and participation in the ICO and supranational meetings related to medical education), we were impressed by the global desire to know what standards exist in other countries for resident performance. Cataract surgery is the most common surgical procedure performed by ophthalmology residents, but no standard competency assessment tool exists either in the United States or elsewhere. In the United States, ophthalmology residency programs require residents to do a specified number of cataract surgeries (quantity), but there are no standard requirements or measures to assess how well the resident did conducting the surgery (quality). Globally, much more variability exists. Indeed, many countries do not require a minimum number of surgeries, let alone standards for competence.

The assessment tool we developed could serve a variety of purposes: (1) It could be internationally applicable, as comments from an international panel of experts were used to adapt it and make it flexible to any phacoemulsification setting. (2) It will decrease subjectivity of the assessment by clearly defining for the assessor what behavior must be observed for each level of proficiency. (3) The rubric will clearly communicate to the learner what is expected to attain competence, and thus this tool can be used for both assessment and teaching.

A good assessment tool must be both valid and reliable. Hence, in this study we investigated the inter-rater reliability of the ICO-OSCAR:phaco. The ACGME suggests assessment tools have an alpha greater than 0.8. Others suggest an alpha of 0.7 is acceptable for a new assessment tool. ^6,27,28 Our data show the ICO-OSCAR:phaco alpha is 0.92. The alpha statistic was lower than 0.7 in three of the 20 individual categories (Capsulorrhexis: Commencement of Flap & Follow-through, Capsulorrhexis: Formation and Circular Completion, and Conjunctival and Corneal Tissue Handling). The wording in these three steps could be further refined to try to improve the alpha. However, the overall alpha is quite good, indicating reliability of the tool as a whole. We attempted to involve investigators and panel reviewers from a variety of countries and continents to ensure global applicability of the tool.

A limitation of the tool is that at first glance the ICO-OSCAR:phaco appears lengthy and perhaps time-consuming. Nevertheless, our experience has been that once used several times, the tool is familiar enough to allow the evaluator to quickly circle the behavior observed in each category and then immediately provide the student with crucial specific and timely formative feedback.

The ICO-OSCAR:phaco has been translated into many languages and is available on the ICO Web site (http://www.icoph.org/resources/230/Surgical-Assessment-Tool-ICO-OSCAR-in-English-Spanish-Chinese-and-Portuguese.html). The ICO is now training physicians to use it when assessing residents and for self-assessment. This is being accomplished by presenting the ICO-OSCAR at international educational ophthalmology meetings and having the audience use the tool to assess short video clips of phacoemulsification procedures. Finally, we plan to produce a CD/DVD containing the tool and rubric, published manuscript, and several surgical videos that have already been scored so that assessors will have examples of how to use the tool and rubric. We will disseminate the CD/DVDs worldwide using the resources of the ICO.

It should be noted that in the United States, ophthalmologists are required to undergo recertification every 10 years. However, currently surgical skill assessment is not part of the recertification effort. Thus, it is inevitable that newer standard-of-care surgical procedures are being performed by ophthalmologists who were never trained to perform them, nor has their competency ever been assessed. Ultimately, it is likely governing bodies will want to assess surgical skills as part of recertification. The ICO-OSCAR:phaco tool could be used for this purpose. Additionally, this tool will allow practicing ophthalmologists the ability to self-assess in a standardized manner.

We subsequently produced similar tools for extracapsular cataract extraction (ICO-OSCAR:ECCE), small incision cataract surgery (ICO-OSCAR:SICS), lateral tarsal strip surgery (ICO-OSCAR:LTS), and strabismus surgery (ICO-OSCAR:strabismus) assessment.^2,29–31 Additionally, we are working on similar tools for panretinal photocoagulation, trabeculectomy, and vitrectomy to produce a selection of globally applicable assessment tools for resident surgical teaching and evaluation.

CONCLUSIONS

The ACGME’s Outcome Project is changing the way physicians are educated and assessed in the United States. The AAO, ABO, AUPO, and ophthalmic educators have combined to meet the ACGME mandate by providing venues for ophthalmic educator collaboration, promoting new teaching and assessment tools, and improving the role of the residency program director. These efforts have spread globally through supranational organizations such as the ICO, PAAO, MEACO, and APAO. International interest has led to the development of new tools designed to teach and assess resident knowledge and skill.

The ICO-OSCAR:phaco is the first internationally applicable valid and reliable tool to be developed.