Association between performance on Neurology In-Training and Certification Examinations

Abstract

Objective:

This study analyzed the relationship between performance on the American Academy of Neurology Residency In-Service Training Examination (RITE) and subsequent performance on the American Board of Psychiatry and Neurology (ABPN) Certification Examination.

Methods:

Pearson correlation coefficients were used to examine the relationship between performance on the RITE and the Certification Examination for 2 cohorts of adult neurologists and 2 cohorts of child neurologists. The 2 cohorts represented test takers for 2008 and 2009.

Results:

For adult neurologists, the correlation between the total RITE and the Certification Examination scores was 0.77 (p < 0.01) in 2008 and 0.65 (p < 0.01) in 2009. For child neurologists, it was 0.74 (p < 0.01) in 2008 and 0.56 (p < 0.01) in 2009.

Discussion:

For 2 consecutive years, there was a significant correlation between performance on the RITE and performance on the ABPN Certification Examination for both adult and child neurologists. The RITE is a self-assessment examination, and performance on the test is a positive predictor of future performance on the ABPN Certification Examination.

The American Academy of Neurology (AAN) Residency In-Service Training Examination (RITE) was developed as a self-assessment instrument that allows trainees to track their knowledge acquisition over the course of their residencies. It has also served as a guide for training directors who can monitor annual performance outcomes to determine the strengths and weaknesses in their educational curricula as well as the effects of any changes made in their programs.

Although the RITE was not designed to be used as a Board examination preparation tool, information on the relationship between performance on the RITE and subsequent performance on the American Board of Psychiatry and Neurology (ABPN) Certification Examination has frequently been sought by both residents and program directors. A study investigating that question found that RITE scores did predict scores on the ABPN Part I Examination.¹ Since then, both the RITE and the ABPN examination have undergone structural changes. The RITE test development committee increased the number of content areas and placed more emphasis on the development of clinically relevant questions.

The ABPN certification process has also undergone structural changes. Prior to 2008, candidates had to pass a multiple-choice examination and an oral examination. With the inception of the 2008 ABPN Certification Examination, the format changed to a computer-administered multiple-choice examination exclusively.

Because of these changes, the AAN and ABPN agreed to conduct another study to determine if senior residents' performance on the RITE was predictive of performance on the ABPN Certification Examination.

METHODS

Description of examinations

Both the RITE and the ABPN Certification Examination were developed and reviewed by committees of subject matter experts, including psychiatrists and test development specialists. Both tests were given under secure conditions following standardized test administration procedures, and results were reported as percent correct scores.

RITE

The 2008 RITE was administered in March to 2,049 participants, of whom 673 were senior residents in US neurology and child neurology residency programs who were preparing to graduate in the summer of 2008. The examination consisted of 437 questions distributed as follows: anatomy (10%), behavioral/psychiatry (11%), clinical adult (16%), clinical pediatrics (10%), contemporary issues (3%), neuroimaging (14%), pathology (11%), pharmacology/chemistry (10%), and physiology (15%). A technical summary of the examination results was prepared for the AAN.²

The 2009 RITE was administered in February to a total of 2,131 participants, of whom 684 were senior residents in US neurology and child neurology residency programs who were preparing to graduate in the summer of 2009. The examination consisted of 443 questions based on the same test blueprint used in 2008. A technical summary of the examination results was prepared for the AAN.³

ABPN Certification Examination

The 2008 ABPN Certification Examination was administered in November to 439 examinees. Of these, 387 were adult neurologists and 52 were child neurologists. The examination had 400 items grouped into 3 components: basic neuroscience (120 items); behavioral neurology, cognition, and psychiatry (80 items); and adult and child clinical neurology (200 items). To pass, the examinees had to score at or above the pass/fail standard set for the total test; the 3 components did not have separate pass/fail standards.

The 2009 ABPN Certification Examination was administered in September/October to 472 new adult neurology candidates and 71 new child neurology candidates. The examination had 400 items that were similarly distributed among the 3 subject areas as the 2008 examination. Of the 200 clinical neurology items, 132 were administered to both the adult and child neurologists. Each group had 68 additional items unique to their specialty.

Subjects

The AAN and the ABPN agreed to collaborate in this study, and RITE examinees signed a release statement when they sat for the examination to have their scores included in the analysis. Their scores were forwarded to the ABPN, and the names of these individuals were matched with the names of the examinees who took the Certification Examinations. As part of the ABPN application process, the applicants signed a statement agreeing to let the Board release information about examination results and examination scores, provided that such data were reported in the aggregate.

Data analysis

Cronbach α or Kuder-Richardson Formula 20 was used to determine the internal consistency reliability of the test scores. (They are equivalent for dichotomously scored items.) In addition to descriptive statistics, Pearson correlation coefficients were computed to examine the relationship between scores on the 2 tests. Because adult neurologists and child neurologists took somewhat different ABPN Certification Examinations in 2009, the results are reported separately for the 2 groups. Expectancy tables were developed to further demonstrate the relationship between the 2 examinations.

RESULTS

The internal consistency reliability coefficients were 0.96 for the 2008 RITE and 0.95 for the 2009 RITE. They were 0.91 for the 2008 ABPN Certification Examination and 0.92 for both of the 2009 examinations. Reliabilities of 0.90 or greater are frequently characterized as indicating consistent measurement.⁴

As indicated in table 1, 364 (63%) of the 581 neurology residents who took the 2008 RITE took the 2008 ABPN examination. Of the 588 senior neurology residents who took the 2009 RITE, 372 (63%) subsequently took the 2009 ABPN examination. Of the 92 senior child neurology residents who took the 2008 RITE, 46 (50%) took the 2008 ABPN examination. Of the 96 senior child neurology residents who took the 2009 RITE, 49 (51%) took the 2009 Certification Examination. The RITE mean scores and the ABPN Certification Examination mean scores and pass rates for all 4 groups appear in table 1, as do the same data for all examinees on both tests.

Table 1.

Summary of RITE and Certification Examination results for adult and child neurologists: 2008–2009

For the neurologists, the correlation between the RITE total score and the Certification total score was 0.77 (p < 0.01) in 2008 and 0.65 (p < 0.01) in 2009. That is, performance on the RITE accounted for 59% and 42% of the variance in the Certification scores, respectively.

For the child neurologists, the correlation between the RITE total score and the Certification total score was 0.74 (p < 0.01) in 2008 and 0.56 (p < 0.01) in 2009. That is, performance on the RITE accounted for 55% and 31% of the variance in the Certification scores, respectively.

Expectancy tables were developed to further illustrate the relationship between performance on the 2 examinations, and the results appear in table 2 for neurologists and in table 3 for child neurologists. Overall, because these examinees had a high pass rate on the ABPN Certification Examination (89%–93%), there is not much differentiation among the RITE score groups, except for those scoring about 1 SD below the mean.

Table 2.

Expectancy table showing percentage of neurologists in each RITE score category who passed the Certification Examination

Table 3.

Expectancy table showing percentage of child neurologists in each RITE score category who passed the Certification Examination

DISCUSSION

The RITE's major purpose has always been to serve as a self-assessment tool for the neurology resident, and analyses of performance by year of training clearly demonstrate that there is a linear progression in knowledge acquisition throughout the 4 years of training.^2,3 In addition, the results of this study demonstrate that performance on the RITE predicts performance on the ABPN Certification Examination. For the years studied, the correlations between the RITE total scores and the Certification total scores ranged from 0.56 to 0.77 and were all significant. Correlations in this range (0.60–0.80) are commonly characterized as strong.⁵

A limitation of the study is that not all those who took the RITE sat for the subsequent Certification Examination. However, the performance of the study groups was very similar to that of all examinees on both examinations. This does raise the issue of the percentage of graduates who typically seek certification as soon as possible after completion of residency. In 2008, 472 neurologists and 77 child neurologists completed training,⁶ and 515 neurologists and 77 child neurologists completed training in 2009.⁷ Hence, it can be estimated that in 2008 82% (387/472) of new neurology graduates sat for the Certification Examination, and in 2009 this increased to 92% (472/515). In 2008, 68% (52/77) of new child neurology graduates sat for the Certification Examination, and in 2009 this increased to 92% (71/77). This suggests that a number of graduates may have opted not to take the first offering of the new Certification Examination. Other reasons for delaying certification may include failure to successfully complete the residency on time; participation in fellowship training, relocation, financial, or other personal hardship; lack of confidence; or simply choosing not to become board certified.

There are various sources of validity evidence for examinations,⁸ and this study addresses several for the RITE and the ABPN Certification Examination, including content outlines reflecting a broadly agreed upon knowledge base for neurology and child neurology; quality items produced by teams of content experts and test development specialists; quality control in administering, collecting, and analyzing test results; high internal consistency reliabilities; improvement in RITE scores with years of training; and, the focus of this study, significant positive relationships between 2 independent measures of similar content.

GLOSSARY

AAN

American Academy of Neurology

ABPN

American Board of Psychiatry and Neurology

RITE

Residency In-Service Training Examination

AUTHOR CONTRIBUTIONS

Dr. Juul was involved in the design of the study, the analysis and interpretation of the data, and drafting and revising the manuscript for content. Dr. Flynn was involved in the design of the study, the analysis and interpretation of the data, and drafting and revising the manuscript for content. Dr. Gutmann was involved in the interpretation of the data and drafting and revising the manuscript for content. Dr. Pascuzzi was involved in the interpretation of the data and drafting and revising the manuscript for content. Dr. Webb was involved in the interpretation of the data and drafting and revising the manuscript for content. Dr. Massey was involved in the interpretation of the data and drafting and revising the manuscript for content. Dr. DeKosky was involved in the interpretation of the data and drafting and revising the manuscript for content. Dr. Foertsch was involved in the interpretation of the data and drafting and revising the manuscript for content. Dr. Faulkner was involved in the interpretation of the data and drafting and revising the manuscript for content.

STUDY FUNDING

No targeted funding reported.

DISCLOSURES

D. Juul and F. Flynn report no disclosures. L. Gutmann receives research funding from Allergan as site principal investigator and NIH grants 1 U01 NS069498-01 A1, 3P50NS044283-07S1, 1 U01 NS026835-01A1, 1 U01 NS058728-01, and 1 U01NS077179-01. R. Pascuzzi and J. Webb report no disclosures. J. Massey received unrestricted education grants from Allergan for the Annual Duke Advanced EMG and Chemodenervation Workshop (2003–2011) and for fellowship funding (2011). She received an LLC-unrestricted education grant from Merz Pharmaceuticals (2011–2012). She is the site principal investigator for an NIH-funded international multicenter Thymectomy Trial in Non-thymomatous Myasthenia Gravis (2006-present). She is the principal investigator for a phase II, double-blind, randomized, 3-way crossover, placebo-controlled, pharmacodynamic study of CD2017357 in patients with generalized myasthenia gravis on standard therapy (Cytokinetics and NINDS IRC3NS070670-01; 2011–present). S. DeKosky is on the Academic Health Center-Pharma Relationships Advisory Board for Pfizer. He is a consultant on drug development for Elan/Wyeth, Genzyme, Helicon Therapeutics, Janssen, Lilly, Merck, Novartis, and Pfizer. None of these interactions provide more than $10,000 per year. All but one (Helicon) are less than $5,000. He is the site principal investigator at the University of Virginia Memory Disorders Clinics for experimental therapeutic trials for Alzheimer's disease medications for Baxter Pharmaceuticals, Elan Pharmaceuticals, Janssen Pharmaceuticals, Novartis Pharmaceuticals, and Pfizer Pharmaceuticals. M. Foertsch and L. Faulkner report no disclosures. Go to Neurology.org for full disclosures.