Administration and performance on the Spinal Cord Injury Medicine Certification Examination over a 10-year period

Abstract

Context/Objective: The examination for Spinal Cord Injury (SCI) Medicine subspecialty certification has been administered since 1998, but published information about exam performance or administration is limited.

Design: Retrospective review

Setting/Participants: We examined de-identified information from the American Board of Physical Medicine and Rehabilitation (ABPMR) database for characteristics and performance of candidates (n = 566) who completed the SCI Medicine Examination over a 10-year period (2005–2014), during which the exam outline and passing standard remained consistent.

Interventions: Not applicable

Outcome Measures: We analysed candidate performance by candidate track, primary specialty, number of attempts, and domains being tested. We also examined candidate perception of the SCI Medicine Exam by analysing responses to a survey taken after exam completion.

Results: Thirty-six percent of candidates who completed the exam during the study period took it for initial certification (23% in the fellowship track and 13% in the practice track offered during the initial “grandfathering” period) and 64% took it for maintenance of certification (MOC) in SCI Medicine. Factors associated with better exam performance included primary specialty certification in Physical Medicine and Rehabilitation (PM&R) and first attempt at passing the exam. For PM&R candidates, ABPMR Part I Examination scores and SCI Medicine Examination scores were strongly correlated. Candidate feedback about the exam was largely positive with 97% agreeing or strongly agreeing that it was relevant to the field and 90% that it was a good test of their knowledge.

Conclusion: This study can inform prospective candidates for the SCI Medicine Examination as well as those guiding them. It may also provide useful information for future exam development.

Introduction

Spinal Cord Injury (SCI) Medicine is a subspecialty approved by the American Board of Medical Specialties (ABMS) that addresses prevention, diagnosis, treatment, and management of traumatic spinal cord injury and non-traumatic etiologies of spinal cord dysfunction.^1,2 The SCI Medicine Examination has been administered since 1998. It was the first subspecialty certification exam offered by the American Board of Physical Medicine and Rehabilitation (ABPMR).³ Timeline of key events related to certification in SCI Medicine is summarized in Table 1.

Table 1. Timeline of key events in SCI Medicine certification.

Year	Key Event
1995	SCI Medicine recognized as an approved subspecialty by ABMS
1996	ACGME approved program requirements for SCI Medicine fellowship
1998	First SCI Medicine Examination administered for initial subspecialty certification
2005	First MOC candidates in SCI Medicine admitted to the exam Standard setting repeated
2007	“Grandfathering” period of practice track eligibility for first-time applicants expired
2015	Examination outline modified from a three-class to a two-class outline Standard setting repeated

The SCI Medicine Examination

The SCI Medicine Examination is a proctored test consisting of 280 single-best-answer multiple-choice questions.⁴ An ABPMR examination committee, consisting of experts in the field, develops exam items. Each item undergoes multiple levels of rigorous review and critique before being included in the exam. The examination is constructed according to the SCI Medicine Examination outline available on the ABPMR website. The exam outline was categorized under three domains or Classes until 2015: Types of Myelopathy, Physiological Complications Due to SCI, and Clinical Decision-Making (Table 2). The outline was modified in 2015, changing item categorization under two instead of three Item Classes (“Type of Myelopathy” and “Pathophysiology, Evaluation, and Management”).⁵

Table 2. SCI Medicine Examination outline by item class (2005–2014).*.

Class 1 (Type of Myelopathy)

- Traumatic

- Non-traumatic

- Non-specified

Class 2 (Physiological Complications Due to SCI)

- Cardiovascular

- Pulmonary

- Genitourinary/Gastrointestinal

- Musculoskeletal

- Neurological/Neuromuscular

- Integumentary/ Systemic

- Cognitive/Psychologic/ Pain/ Non-specified

Class 3 (Clinical Decision-Making)

- Patient Evaluation and Diagnosis/ Electrodiagnosis

- Patient Management

- Basic and Clinical Sciences

*The SCI Medicine Examination outline was modified in 2015, with a change in item categorization from three to two Item Classes (“Type of Myelopathy” and “Pathophysiology, Evaluation, and Management”). The current exam outline is available at www.abpmr.org.

Standard setting is conducted periodically to establish passing standards for the exam. During standard setting, a group of experts in SCI Medicine from a broad range of SCI experience, training, practice type, and location engages in a multi-step process. They are tasked with determining the minimum amount of knowledge needed to become certified, based on their understanding of the field, the purpose of the exam, and the content being tested. This method is commonly used in high stakes examinations.⁶ It establishes a pass/fail point (passing standard) that is applied uniformly across the exam in different years and is independent of the level of difficulty or the ability of the cohort taking the exam in a particular year.^7,8 The candidate’s raw score (number of items answered correctly) is transformed into a “scaled score” that takes into account year to year variations in the exam forms, including form difficulty. Standard setting was done in 2005 and repeated in 2015. The passing standard for the exam during this period was a scaled score of 450.

The exam was, therefore, largely uniform between 2005 and 2014, following the same outline and passing standards throughout that period.

SCI Medicine subspecialty eligibility requirements

Subspecialty certification in SCI Medicine is offered to diplomates of all ABMS member boards, from specialties related to the care of persons with SCI. Applicants must be current diplomates in good standing of a member board of the ABMS and have a current, valid, and unrestricted license to practice medicine in at least one jurisdiction in the United States, its territories, or Canada.⁴

Eligibility criteria for taking the exam until 2007 included either successful completion of 12 months of an SCI Medicine fellowship accredited by the Accreditation Council for Graduate Medical Education (ACGME), or meeting practice requirements with a minimum of three years of practice primarily in SCI Medicine beyond completion of residency in the primary specialty.³ The latter “grandfathering” period of practice track eligibility for first-time applicants expired in 2007. After 2007, first-time applicants are required to successfully complete 12 months of an ACGME-accredited SCI Medicine fellowship following residency in the primary specialty, and obtain a recommendation by the SCI Medicine fellowship program director for admissibility to take the subspecialty SCI Medicine Examination. The applicant must complete the training program on or before August 31^st of the year of the scheduled examination.⁴

Maintenance of certification in SCI Medicine

Upon successful completion of the examination, the ABPMR grants a subspecialty certificate in SCI Medicine stating that the candidate has met the requirements for certification. The certificate is a 10-year, time-limited certificate for which maintenance of certification (MOC) is necessary. The certificate expires on December 31^st of the tenth year of the cycle. MOC includes achieving a passing score on the SCI Medicine Examination prior to the certificate expiration date. The examination may be taken in years 7–10 of the SCI Medicine MOC cycle.⁴ The first SCI Medicine certificates were issued in 1998, so 2005 was the first year that diplomates could take the exam for MOC in SCI Medicine. From 2005 onwards, therefore, candidates for SCI Medicine Examination include both those applying for initial SCIM certification and those taking the exam for MOC in SCI Medicine.

This study examined administration and performance on the SCI Medicine Examination including candidate characteristics, performance, factors related to performance, and candidate perception of the exam.

Methods

We conducted a retrospective analysis of de-identified information from the ABPMR database relating to the SCI Medicine Examination over a 10-year period (2005–2014), during which the exam outline and passing standard remained consistent. We examined the numbers and characteristics of candidates taking the SCI Medicine Examination; evaluated candidate performance (pass rates and mean scaled scores); and analyzed exam performance by candidate track, primary specialty, number of attempts, and domains being tested. For fellowship candidates, we studied association between exam performance and delay in taking the exam following completion of SCI fellowship.

We evaluated psychometric characteristics of the exam including item reliability, i.e. ability to distinguish satisfactory from non-satisfactory performance in a reproducible manner. We also examined candidate perception of the SCI Medicine Examination by analyzing responses to a de-identified survey taken by candidates immediately after completing the exam. The candidate survey was changed to a 4-point scale (Strongly Agree, Agree, Disagree, Strongly Disagree) in 2007, so we excluded earlier years and evaluated survey data only from 2007–2014.

The Institutional Review Board of the VA Boston Health Care System approved the study protocol. SPSS software was used for all analyses.

Results

Candidates

A total of 566 candidates took the SCI Medicine Examination between 2005–2014. Two hundred and four (36%) of these took the exam for initial certification and 362 (64%) for maintenance of certification (MOC) in SCI Medicine. Of those taking the exam for initial certification, 131 had completed an ACGME-accredited SCI fellowship and 73 were eligible through the practice track (Table 3).

Table 3. SCI Medicine exam candidate characteristics, and performance (2005–2014).

	Number (% of total)1	Pass Rate (%)2	Mean Scaled Score2,3
All candidates	566	83	487
Type/Track
Initial certification	204 (36%)
Accredited fellowship	131 (23%)	92	497
Practice Track	73 (13%)	82	484
MOC candidates	362 (64%)	87	493
Number of attempts
First	504 (89%)	87*	492*
Second	51 (9%)	51*	447*
Three or more	11(2%)	27*	439
Primary specialty
PMR	508 (90%)	89*	500*
Non-PMR	58 (10%) (28 ABIM, 15 ABPN, 5 ABFM, 10 Other)	73*	471*

¹Percent rounded to closest number; ²Pass rate and mean scaled scores by candidate type/track and by primary specialty are for initial attempts only, not repeat attempts; ³Mean scaled scores are on a scale of 200–800; the passing standard during the study period was a scaled score of 450 (see text for the definition of scaled score).

MOC, Maintenance of Certification; PMR, Physical Medicine and Rehabilitation; ABIM, American Board of Internal Medicine; ABPN, American Board of Psychiatry and Neurology; ABFM, American Board of Family Medicine.

*Significant, P < 0.001.

Five hundred and eight (90%) of candidates for subspecialty SCI Medicine certification had primary certification in Physical Medicine and Rehabilitation (PMR), while 58 (10%) had other primary Board certification (most commonly Internal Medicine or Neurology). Sixty-two (11%) of the 566 were repeat candidates who had failed the exam in previous attempts.

Candidate Performance

Overall pass rate during the 10-year study period was 83% (Table 3). Mean Scaled Score was 487 (On a scale of 200 to 800).

Performance by candidate track

Fellowship candidates had a higher pass rate (92%) on initial attempt than those in practice track (82%) and MOC (87%), and had higher mean scores, but the differences did not reach statistical significance (Table 3).

Performance by primary specialty

PM&R candidates performed significantly better on initial attempt (89% pass rate) than candidates with non–PMR Board certification (73% pass rate) (P < 0.01). A similar trend was noted for mean scaled scores between the two groups (Table 3).

For PM&R candidates, SCI Medicine Examination scores on initial attempt correlated significantly with scores on initial attempt on the ABPMR Part 1 Examination for primary certification in PM&R. Part I scores accounted for 34.1% of the variance in SCI Medicine scores (P < 0.0001) (Figure 1).

Figure 1.

Correlation between SCI Medicine and PMR Part I Examination scores. For PMR candidates, there was a strong correlation between scaled scores for SCI Medicine Examination scores and Part I Examination. Part I scores accounted for 34.1% of the variance in SCI Medicine scores (P < 0.0001). Scores for both the SCI Medicine and the PMR Part I examinations are for the initial attempt only, even for those with multiple attempts for either exam. The cut-offs for the passing standard for the SCI Medicine Examination (a scaled score of 450) and the passing standard for the Part 1 Examination (a scaled score of 420) are also shown. See text for the definition of scaled scores.

Performance by item class

Fellowship candidates had significantly higher mean sub-scores than MOC candidates for items classified as Traumatic SCI, Pulmonary, Genitourinary/Gastrointestinal, Musculoskeletal, and Neurological/Neuromuscular (P < 0.01) (Table 4). “Pulmonary” items had the lowest sub-scores for both groups. PMR candidates had significantly higher sub-scores than non-PMR candidates for items categorized as Traumatic SCI in Class I, and for Pulmonary, Musculoskeletal, Neurological, and Cognitive/Psychological items in Class II (P < 0.01).

Table 4. Performance by item class.¹ .

Item Class	Mean Scaled Sub-Score
	Fellowship (N = 120)	MOC candidates (N = 327)
Class 1 (Type of Myelopathy)
- Traumatic*	6.9	6.5
- Non-traumatic	6.7	6.6
- Non-specified	7.0	6.8
Class 2 (Physiological Complications Due to SCI)
- Cardiovascular	6.8	6.6
- Pulmonary*	6.6	6.1
- Genitourinary/Gastrointestinal*	7.1	6.8
- Musculoskeletal*	6.9	6.6
- Neurological/Neuromuscular*	6.7	6.4
- Integumentary/ Systemic	7.0	6.8
- Cognitive/Psychologic/ Pain/ Non-specified	6.7	6.6
Class 3 (Clinical Decision-Making)
- Patient Evaluation and Diagnosis/ Electrodiagnosis*	7.0	6.6
- Patient Management	6.7	6.6
- Basic and Clinical Sciences*	6.8	6.4

¹Initial attempts only, excludes repeat attempts

* Significant, P < 0.01

Performance by number of attempts

Performance declined significantly with repeat attempts after failing on initial attempt. Pass rate was 87% with the first attempt, 51% with the second attempt, and 27% with 3 or more attempts (P < 0.001), and there was a significant decline in mean scores between the first and second attempts. (Table 3).

Performance by time since completing fellowship

Fellowship candidates who waited one or more years after fellowship to take the exam performed worse overall than those taking the exam within the first year of completing fellowship (Table 5). However the difference was only significant for those whose SCI Medicine Examination delay was associated with failure to pass Part I or Part II of the primary PMR certification exam in the first attempt.

Table 5. Association of exam performance with delay in taking exam after SCI Medicine fellowship.

Delay	N (%)	Mean Scaled Score	Pass Rate
0 (same year)	65 (54%)	505	97%
1+ years
• No prior fails	42 (35%)	489	88%
• Failed Part I, Part II, or both	13 (11%)	485	77%*

*Post hoc tests reveal that only the difference in pass rates between no delay and delay with prior fails is significant (P < 0.01).

Exam item reliability

Despite small sample sizes, exam item reliability (i.e. ability to distinguish satisfactory from non-satisfactory performance in a reproducible manner) was in the Good to Excellent range, ranging between 0.85 and 0.90. These values indicate that variability in scores was largely due to differences in true ability of candidates.

Candidate perception of the examination

Candidate feedback about the exam, in response to items surveyed on a 4-point scale, was largely positive (Table 6). Ninety seven percent agreed or strongly agreed that the exam was relevant to the field and 90% that it was a good test of their knowledge. There was no significant difference in perception of the exam for survey items between candidates for initial certification and MOC candidates.

Table 6. Candidate perception of the SCI Medicine Exam (percent who “Agree” or “Strongly Agree”).

Item	Initial candidates	MOC candidates
The examination was relevant to the field	97	97
The examination was a good test of your knowledge	92	89
The examination content reflected your scope of training	77	79
The level of difficulty of the examination was appropriate	78	76
The examination was fair	81	80

Discussion

There is very little information about SCI Medicine Examination performance or administration in published literature. In this study we examined the characteristics of candidates who took the SCI Medicine Examination for subspecialty certification between 2005 and 2014, and investigated the association between candidate factors and exam performance. Of the 566 candidates who completed the exam during the 10-year study period, 23% took it for initial certification after completing an accredited SCI Medicine fellowship, 13% for initial certification after meeting practice track eligibility qualifications in the “grandfathering” period, and 64% took the exam for MOC in SCI Medicine.

Factors associated with significantly better exam performance included primary specialty certification in PM&R and first attempt at passing the exam. Although completion of an ACGME-accredited SCI Medicine fellowship was associated with a higher pass rate and mean scaled scores, the difference did not reach statistical significance. For candidates with PM&R primary certification there was a strong correlation between scores in ABPMR Part 1 Exam for certification in PM&R and SCI Medicine Examination scores. This finding is consistent with other reports of strong correlation between scores of different licensing, certification, and in-training examinations with multiple choice items across medical training, indicating that good test taking skills appear to translate across different multiple choice exams.⁹

The decline in performance with repeat attempts, noted in our study, has also been reported for other certification exams.¹⁰ Since performance declines significantly with repeat attempts, those who don’t pass the SCI Medicine Examination on first attempt should give serious thought to additional strategies for exam preparation, as should PMR candidates whose score on their initial attempt for the ABPMR Part I Certification examination was below the passing standard for that exam (Figure 1). Our findings can also provide guidance about specific exam domains that may require extra preparation, e.g. for MOC candidates and those from non-PMR primary specialties. When assessing performance by item class, “Pulmonary” items had the lowest mean scaled scores overall. MOC candidates had significantly lower sub-scores for Traumatic SCI, Pulmonary, Genitourinary/Gastrointestinal, and Musculoskeletal item classes than initial certification candidates with an SCIM fellowship. Pulmonary impairment only impacts cervical and high thoracic injuries so there is likely less consistent experience with these issues in practice settings compared to fellowship programs, especially in the acute and immediate post-acute phases of injury. Greater emphasis on pathophysiology of genitourinary and gastrointestinal impairments during fellowship may possibly explain better test scores for fellowship candidates in those domains. Candidates from non-PMR primary specialties had significantly lower sub-scores than PMR candidates for Traumatic SCI, Pulmonary, Musculoskeletal, Neurological, and Cognitive/Psychological items.

Fellowship candidates who delayed taking the exam for one or more years after completing fellowship performed worse overall than those taking the exam in the first year after fellowship. This is consistent with findings in some other studies that examined impact of delay in taking certification examination after training completion.¹¹ However in our study we found that the difference was significant only for those whose SCI Medicine Examination delay was associated with failure to pass Part I or Part II of the primary specialty exam in PM&R in the first attempt. Fellowship programs may consider providing additional support to help fellows prepare for primary certification exam, thus helping them meet eligibility criteria for taking the SCI Medicine Examination at the earliest opportunity after fellowship completion.

An important consideration when assessing the psychometric characteristics of an examination is the reliability of scores and the reproducibility of pass/fail decisions. Reliability and reproducibility values for the exam met industry standards and indicated that variability in scores was largely due to differences in true ability of candidates.¹² We also found an overall positive perception of the exam among candidates, especially regarding its relevance to the field and as a good test of their knowledge. Moreover it was reassuring that the largely positive perception of the exam was true both for those taking the exam for initial certification and those taking it for MOC. There was no significant difference between the two groups in the positive response rate to survey items. This suggests that despite having the same exam for the two groups at different points in their career, it may be meeting the needs of both groups.

The positive candidate feedback and exam reliability findings in our study likely reflect the rigor and care that goes into item development for the SCI Medicine Examination. Exam items are developed by clinicians with expertise in the content and scope of SCI Medicine and training in item writing. Before being accepted to the exam, items undergo multiple series of reviews not only for accuracy, clarity, and completeness but also to ensure that the content reflects core knowledge and relevance to SCI Medicine. Items continue to undergo periodic reviews with elimination of items that are identified as flawed or no longer reflecting evolving practice standards.

One candidate survey item that does raise some questions was the item asking whether examination content reflected the candidate’s scope of training. It had a relatively low positive response even among those initially certifying, with only 77% agreeing or strongly agreeing. Since the exam outline closely follows ACGME requirements for fellowship training in SCI Medicine, fellowship directors may consider reviewing whether fellows in their program are consistently exposed to the full scope of SCI Medicine and identify ways to supplement training for areas with inadequate exposure. Similarly, when preparing for the exam, candidates should keep in mind that the SCI Medicine Examination is designed to assess the full scope of the specialty including the full range of traumatic and non-traumatic myelopathies, even though individual scope of practice may be more limited. These differences between the overall scope of SCI Medicine as a subspecialty and the scope of typical SCI Medicine training or practice are also pertinent considerations for future exam development.

There are a number of limitations to our study. Even though data was pooled over a 10 year period, the relatively small number of candidates limits the robustness of statistical analyses, especially for sub-groups. Moreover, while the exam outline and passing standard was consistent over the study period, current findings may not necessarily apply to future exam administration and performance as the exam continues to evolve over time.

Conclusions

This study can inform prospective candidates for the SCI Medicine Examination as well as those guiding them. It may also provide useful information for future exam development, and assist in efforts to continue to enhance certification-related assessments.

Disclaimer statements

Contributors None.

Funding None.

Conflicts of Interest Two authors are American Board of Physical Medicine and Rehabilitation (ABPMR) Board Directors. One author is employed by the ABPMR as a psychometrician. Contents of this article do not represent the views of the Department of Veterans Affairs or the United States Government.

Ethics Approval None.

Disclosures: Drs. Chiodo and Sabharwal are American Board of Physical Medicine and Rehabilitation (ABPMR) Board Directors. Dr. Raddatz is employed by the ABPMR as a psychometrician.

ORCID

Mikaela M. Raddatz http://orcid.org/0000-0001-8086-6784