Residency program characteristics that are associated with pass rate of the American Board of Pediatrics certifying exam

Abstract

Background

The US is home to almost 200 pediatrics residency programs; despite this, there is little information about the relationship between program characteristics and performance in the American Board of Pediatrics (ABP) certifying exam.

Objective

To evaluate the relationship between pass rate of the ABP certifying exam with the characteristics of categorical pediatrics residency programs.

Methods

This retrospective, cross-sectional study used publicly available data from the ABP website and the Fellowship and Residency Electronic Interactive Database. All programs that reported pass rates were included. The analysis, comprising univariate and multivariate linear regression, involved determining how 69 factors (eg, general information, number of faculty and trainees, work schedule, educational environment) related to the pass rate.

Results

Of 199 programs, 194 reported pass rates. The univariate analysis revealed 20 program characteristics with P-values <0.10. However, in the multivariate analysis, pass rate was significantly associated with only three program characteristics: ratio of full-time equivalent paid faculty to positions, percentage of US medical graduates, and average hours per week of regularly scheduled lectures or conferences.

Conclusion

Unlike in previous studies, location and program size were not significantly associated with the pass rate in this multivariate analysis. The finding regarding the ratio of full-time equivalent paid faculty to positions highlighted the benefits of a well-supervised training environment, while that regarding the percentage of US medical graduates indicated the necessity of high competition in residency programs. Finally, longer hours per week of regularly scheduled lectures or conferences were associated with better academic outcomes, both statistically and intuitively.

Introduction

In the US, pediatrics residency programs are the third largest medical specialty in terms of the number of residency positions; in 2015 alone, there were 3,936 applicants for 2,668 pediatrics residency program positions.¹ Before being able to practice as a pediatrician, a physician must pass the American Board of Pediatrics (ABP) certifying exam. In the US, between 2012 and 2014, the average pass rate for first-time takers of the ABP certifying exam was 86.74%.² While this rate is relatively high, those who fail the exam can only retake it a year later. Additionally, the Accreditation Council for Graduate Medical Education (ACGME) requires pediatrics residency programs to achieve a 70% pass rate for first-time takers. Thus, failing the board certifying exam has a considerable negative impact on the track records for residency programs.³

The majority of observational studies in various medical specialties have shown that board certification is associated with quality of care.^4–7 Thus, finding ways to improve the pass rate of the board certifying exam could boost the reputations of residency programs as well as improve patient outcomes at training centers. Unfortunately, there is limited information on what program characteristics best influence the pass rate of the ABP certifying exam. Previous investigations on this topic showed that program location and size, United States Medical Licensing Examination (USMLE) Step 1 scores, city population, and whether the program was a dual internal medicine–pediatrics program were significantly related to the pass rate.^8–11 Performance on in-training examinations was not significantly related.¹² Similar results were found in a study of factors affecting the pass rates of the American Board of Family Medicine certifying exam, in which the significantly related factors were program location and size, accreditation cycle length, opportunities for international experiences, and training in alternative medicine.¹³

The current study aimed to evaluate the relationships between the pass rate of the ABP certifying exam and the characteristics of 3-year categorical pediatrics residency programs. A better understanding of these relationships will benefit various parties, including program directors, who will be able to improve the educational quality of pediatrics residency training; residency candidates, who will be able to make a better career decisions; and patients, who will obtain a higher quality of care from better trained pediatricians.

Methods

Using a retrospective, cross-sectional design, this study evaluated the relationships between the pass rate of the ABP certifying exam and most of the program characteristics available in the Fellowship and Residency Electronic Interactive Database (FREIDA^®). This study covered all 3-year categorical pediatrics residency programs in the US, including the US territory of Puerto Rico. The program characteristics of these categorical pediatrics residency programs were extracted from FREIDA^® on May 9, 2015, using a computerized automation program; there were 69 such characteristics in total. The 2012–2014 pass rates of the ABP certifying exam, which were the most recent rates at the time of study, were obtained from the ABP website.² The only exclusion criterion for residency programs was not reporting the pass rate.

Of the 69 characteristics, six dealt with location and general information, six with the number of faculty and trainees, six with work schedule, three with the educational environment, 17 with educational benefits, seven with educational features, five with resident evaluation, four with program evaluation, 13 with employment policies and benefits, and two with compensation and leave. The majority of these variables were dichotomous, while the rest were either continuous or categorical.

Regarding location, the pediatrics residency programs were grouped by the ten regions reported in FREIDA^® (Table 1). Program size was calculated using the average number of residency positions from postgraduate years 1–3. Two types of exam scores are commonly used as requirements for interviews: the USMLE and the Comprehensive Osteopathic Medical Licensing Examination of the United States. However, because the majority of residency programs used only USMLE scores as a requirement for inviting candidates for interviews, only USMLE score requirements were included as a characteristic in the analysis for this study. I did not consider the number of faculty members in each program, because a more meaningful measure, the ratio of full-time equivalent paid faculty to positions (faculty-to-position ratio), was available. Other non meaningful or hard-to-quantify characteristics (eg, visa status of international medical graduates, major medical benefits, sick days, call schedules, and average score requirement) were similarly excluded.

Table 1.

Regional locations of pediatrics residency programs

Regional location	State	Number of programs with pass rate	Mean ± SD of pass rate (%)
Mid-Atlantic	Delaware, New Jersey, New York, Pennsylvania	47	80.20±11.46
South Atlantic	District of Columbia, Florida, Georgia, Maryland, North Carolina, South Carolina, Virginia, West Virginia	34	84.25±12.85
East North Central	Illinois, Indiana, Michigan, Ohio, Wisconsin	33	84.84±13.32
Pacific	California, Hawaii, Oregon, Washington	20	89.70±7.52
West South Central	Arkansas, Louisiana, Oklahoma, Texas	19	82.94±9.20
West North Central	Kansas, Minnesota, Missouri, Nebraska, South Dakota,	12	86.29±10.96
New England	Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, Vermont	11	91.98±7.81
East South Central	Alabama, Kentucky, Mississippi, Tennessee	9	84.81±9.04
Mountain	Arizona, Colorado, Nevada, New Mexico, Utah	6	85.17±13.25
Territory	Puerto Rico	3	45.02±33.78
All locations		194	83.82±12.92

All statistical analyses were conducted with STATA version 13.0 (StataCorp, College Station, TX, USA). All categorical variables were dichotomized for the purposes of statistical analyses.

The main assumption of this study was that the data would form a normal distribution at the population level. As such, data were presented in terms of mean ± SD for continuous variables, and number (%) for dichotomous variables. In the univariate analysis, the relationships between pass rate and all program characteristics were identified by univariate linear regression. Only characteristics with P-values <0.10 in the univariate analysis were entered into the multivariate linear regression analysis, which was intended to identify the significant independent predictors of the ABP exam pass rate. In the multivariate analysis, the significance level α was set at 0.05. Being a retrospective and nonhuman study, this study was exempted from institutional review board (Rajavithi Hospital, Bangkok, Thailand) approval.

Results

According to a FREIDA^® search, at the time of study, there were 199, 3-year categorical pediatrics residency programs throughout the US. Five of these programs were excluded because they did not report the ABP exam pass rate; thus, 194 programs were included in the analysis. The baseline characteristics of all programs are summarized and presented in Table 2. As shown in Tables 1 and 3, the average pass rate of the ABP certifying exam for the entire US was 83.82±12.92. By region, New England programs had the highest average pass rate (91.98±7.81; P=0.031), while programs in Puerto Rico had the lowest average (45.02±33.78; P<0.001).

Table 2.

Baseline characteristics of pediatrics residency programs

Program characteristics	Number of observations	Mean ± SD or n (%)
Location
Mid-Atlantic	194	47 (24.23)
South Atlantic	194	34 (17.53)
East North Central	194	33 (17.01)
Pacific	194	20 (10.31)
West South Central	194	19 (9.79)
West North Central	194	12 (6.19)
New England	194	11 (5.67)
East South Central	194	9 (4.64)
Mountain	194	6 (3.09)
Territory	194	3 (1.55)
General information
Program size	151	15.32±8.46
Program type
University-based	151	82 (54.3)
Community-based university-affiliated hospital	151	64 (42.38)
Community-based	151	5 (3.31)
Offers preliminary positions	151	0.11±0.32
Minimum score of USMLE Step 1 for interview consideration	105	199.23±10.67
Minimum score of USMLE Step 2 for interview consideration	57	204.49±9.48
Faculty and trainee information
Full-time paid female physician faculty (%)	149	47.78±10.95
Ratio of full-time equivalent paid faculty to positions	151	2.34±1.58
US medical graduate (%)	109	60.2±33.97
International medical graduate (%)	109	26.98±30.66
Doctor of osteopathy (%)	109	12.62±15.78
Female	109	72.9±8.96
Work schedule information
Average hours per week on dutya,b	151	62.32±6.5
Maximum consecutive hours on dutya,b	151	16.09±2.13
Average number of 24-hour off duty periods per weekb	151	1.31±0.31
Program allows moonlightingc	151	96 (63.58)
Night float system (in or beyond first year)	151	123 (81.46)
Offers awareness and management of fatigue in residents	151	151 (100)
Educational environment
Average hours per week of regularly scheduled lectures or conferencesb	151	7.25±1.92
Training at hospital outpatient clinicsb	149	0.31±0.13
Training in ambulatory nonhospital community-based settingsb	127	0.12±0.08
Educational benefits
Physician impairment prevention curriculum	151	134 (88.74)
Integrative medicine curriculum	151	17 (11.26)
Debt management or financial counseling	151	136 (90.07)
Formal program to develop teaching skills	151	149 (98.68)
Formal mentoring program	151	150 (99.34)
Formal program to foster interdisciplinary teamwork	151	118 (78.15)
Continuous quality improvement training	151	151 (100)
International experience	151	114 (75.5)
Resident retreats	151	146 (96.69)
Off-campus electives	151	146 (96.69)
Hospice or home care experience	151	106 (70.2)
Cultural competence awareness	151	145 (96.03)
Instruction in medical Spanish or other non-English language	151	55 (36.42)
Alternative or complementary medicine curriculum	151	53 (35.1)
Economics of health care systems curriculum	151	78 (51.66)
MPH or MBA or PhD training	151	31 (20.53)
Required research rotation	150	21 (14)
Educational features
Offers additional training beyond accredited length	151	19 (12.58)
Offers a primary care track	151	66 (43.71)
Offers a rural track	151	8 (5.3)
Offers a women’s health track	151	1 (0.66)
Offers a hospitalist track	151	22 (14.57)
Offers a research track or non-accredited fellowship	151	32 (21.19)
Offers another track	151	57 (37.75)
Resident evaluation
Yearly specialty in-service examination required	151	151 (100)
Patient surveys	151	148 (98.01)
Portfolio system	151	140 (92.72)
360-degree evaluations	151	150 (99.34)
Objective structured clinical examinations (OSCE)	151	97 (64.24)
Program evaluation
Program graduation rates	151	146 (96.69)
Resident assessment of curriculum	151	136 (90.07)
In-training examination scores	151	151 (100)
Performance-based assessment scores	151	110 (72.85)
Employment policies and benefits
Part-time or shared positions	151	14 (9.27)
On-site child care	151	56 (37.09)
Subsidized child care	151	13 (8.61)
Allowance or stipend for professional expenses	151	147 (97.35)
Leave for educational meetings or conferences	151	115 (76.16)
Moving allowance	151	25 (16.56)
Housing stipend	151	12 (7.95)
On-call meal allowance	151	148 (98.01)
Free parking	151	108 (71.52)
Personal digital assistants (PDAs)	151	40 (26.49)
Placement assistance upon completion of program	151	85 (56.29)
Cross coverage in case of illness or disability	151	146 (96.69)
Policy prohibits hiring smokers or users of nicotine products	151	9 (5.96)
Compensation and leave
Salary compensationb (US$)	144	51,544.93±3771.46
Vacation daysb	151	19.05±4.19

Notes:

^aExcluding beeper call;

^bduring first year;

^cbeyond first year.

Abbreviations: MPH, Master of Public Health; MBA, Master of Business Administration; USMLE, United States Medical Licensing Examination.

Table 3.

Results of the univariate linear regression analysis between the pass rate and pediatrics program characteristics

Program characteristics	β-coefficient (standard error)	P-value
Location
Mid-Atlantic	−4.7762 (2.1430)	0.027*
South Atlantic	0.5307 (2.4458)	0.828
East North Central	1.2381 (2.4736)	0.617
Pacific	6.5624 (3.0215)	0.031*
West South Central	−0.9713 (3.1281)	0.757
West North Central	2.6405 (3.8559)	0.494
New England	8.6584 (3.9724)	0.031*
East South Central	1.0376 (4.4209)	0.815
Mountain	1.4005 (5.3709)	0.795
Territory	−39.4021 (6.9801)	<0.001*
General information
Program size	0.4926 (0.1095)	<0.001*
Program type
University-based	5.5210 (1.8400)	0.003*
Community-based university-affiliated hospital	−2.3010 (1.9710)	0.244
Community-based	−13.6438 (5.7858)	0.019*
Offers preliminary positions	5.2527 (3.0852)	0.091
Minimum score of USMLE Step 1 for interview consideration	0.0229 (0.1089)	0.834
Minimum score of USMLE Step 2 for interview consideration	0.1021 (0.1814)	0.576
Faculty and trainee information
Full-time paid female physician faculty (%)	0.0051 (0.0907)	0.956
Ratio of full-time equivalent paid faculty to positions	2.1671 (0.6015)	<0.001*
US medical graduate (%)	0.2217 (0.0271)	<0.001*
International medical graduate (%)	−0.2529 (0.0295)	<0.001*
Doctor of osteopathy (%)	−0.0743 (0.0741)	0.318
Female	0.2043 (0.1296)	0.118
Work schedule information
Average hours per week on dutya,b	−0.1063 (0.1516)	0.484
Maximum consecutive hours on dutya,b	−0.6825 (0.4599)	0.140
Average number of 24-hour off duty periods per weekb	−1.4257 (3.1499)	0.651
Program allows moonlightingc	4.7592 (2.0086)	0.019*
Night float system (in or beyond first year)	−2.3950 (2.5258)	0.345
Offers awareness and management of fatigue in residents	0 (0)	N/A
Educational environment
Average hours per week of regularly scheduled lectures/conferencesb	1.5593 (0.4996)	0.002*
Training at hospital outpatient clinicsb	−1.3065 (7.9874)	0.870
Training in ambulatory non-hospital community-based settingsb	4.4646 (13.1316)	0.734
Educational benefits
Physician impairment prevention curriculum	0.5699 (3.1147)	0.855
Integrative medicine curriculum	0.7154 (3.1145)	0.819
Debt management or financial counseling	−0.8823 (3.2909)	0.789
Formal program to develop teaching skills	−11.0277 (8.5650)	0.200
Formal mentoring program	−13.9993 (12.0850)	0.249
Formal program to foster interdisciplinary teamwork	2.5854 (2.3731)	0.278
Continuous quality improvement training	0 (0)	N/A
International experience	10.5934 (2.1183)	<0.001*
Resident retreats	−4.8533 (5.4883)	0.378
Off-campus electives	10.1226 (5.4399)	0.065
Hospice or home care experience	3.2087 (2.1366)	0.135
Cultural competence awareness	−1.9387 (5.0380)	0.701
Instruction in medical Spanish or other non-English language	1.1077 (2.0441)	0.589
Alternative or complementary medicine curriculum	2.7194 (2.0509)	0.187
Economics of health care systems curriculum	0.2675 (1.9702)	0.892
MPH or MBA or PhD training	−0.5453 (2.4372)	0.823
Required research rotation	−7.1112 (2.7836)	0.012*
Educational features
Offers additional training beyond accredited length	−2.4001 (2.9622)	0.419
Offers a primary care track	0.0629 (1.9850)	0.975
Offers a rural track	2.5795 (4.3906)	0.558
Offers a women’s health track	−17.9019 (12.0503)	0.139
Offers a hospitalist track	1.398 (2.7885)	0.617
Offers a research track or non-accredited fellowship	5.3917 (2.3685)	0.024*
Offers another track	6.9302 (1.9502)	0.001*
Resident evaluation
Yearly specialty in-service examination required	0 (0)	N/A
Patient surveys	1.8739 (7.0541)	0.791
Portfolio system	1.8147 (3.7857)	0.632
360-degree evaluation	−11.9659 (12.0996)	0.324
Objective structured clinical examinations (OSCE)	−1.9645 (2.0479)	0.339
Program evaluation
Program graduation rates	−5.1615 (5.4864)	0.348
Resident assessment of curriculum	0.3442 (3.2916)	0.917
In-training examination scores	0 (0)	N/A
Performance-based assessment scores	−1.6775 (2.2096)	0.449
Employment policies and benefits
Part-time or shared positions	−0.5958 (3.3944)	0.861
On-site child care	3.1378 (2.0221)	0.123
Subsidized child care	4.1467 (3.4937)	0.237
Allowance or stipend for professional expenses	−8.1706 (6.0946)	0.182
Leave for educational meetings or conferences	−0.9100 (2.3095)	0.694
Moving allowance	1.4029 (2.6465)	0.597
Housing stipend	1.2049 (3.6390)	0.741
On-call meal allowance	10.2572 (7.0056)	0.145
Free parking	−1.7787 (2.1768)	0.415
Personal digital assistants (PDAs)	1.4153 (2.2282)	0.526
Placement assistance upon completion of program	3.7371 (1.9612)	0.059
Cross coverage in case of illness or disability	12.9295 (5.3998)	0.018*
Policy prohibits hiring smokers or users of nicotine products	1.3906 (4.1573)	0.738
Compensation and leave
Salary compensationb (US$)	0.0004 (0.0003)	0.128
Vacation daysb	−0.0190 (0.2358)	0.936

Notes:

^aExcluding beeper call;

^bduring first year;

^cbeyond first year;

^*P-value <0.05.

Abbreviations: MPH, Master of Public Health; MBA, Master of Business Administration; USMLE, United States Medical Licensing Examination; N/A, not applicable.

The univariate linear regression was performed to identify program characteristics with P<0.10. As shown in Table 3, there were 20 variables with P<0.10, including location (Mid-Atlantic [β=−4.7762, P=0.027], Pacific [β=6.5624, P=0.031], New England [β=8.6584, P=0.031], and US territory [β=−39.4021, P<0.001]); program size (β=0.4926, P<0.001); program type (university-based [β=5.5210, P=0.003] and community-based [β=−13.6438, P=0.019]); faculty-to-position ratio (β=2.1671, P<0.001); percentage of US medical graduates ([%USMD], β=0.2217, P<0.001); percentage of international medical graduates (β=–0.2529, P<0.001); work schedule allowing moonlighting (β=4.7592, P=0.019); average hours per week (hr/wk) of regularly scheduled lectures or conferences ([average hr/wk of lectures] β=1.5393, P=0.002); offering international experience (β=10.5934, P<0.001); required research rotation (β=−7.1112, P=0.012); offering a research track or non-accredited fellowship (β=5.3917, P=0.024); offering another track (β=6.9302, P=0.001); and cross coverage in case of illness or disability (β=12.9295, P=0.018).

Table 4 shows the multivariate analysis. Of the 20 variables included, only three were significant independent predictors: faculty-to-position ratio (β=1.3919, P=0.008); %USMD (β=0.2054, P<0.001); and average hr/wk of lectures (β=1.1987, P=0.008). Altogether, these three predictors explained 44.83% of the variance in pass rate.

Table 4.

Multivariate linear regression of the ABP pass rate and significantly associated program characteristics

Program characteristics	β-coefficient (standard error)	P-value
Ratio of full-time equivalent paid faculty to positions	1.3919 (0.5146)	0.008
US medical graduate (%)	0.2054 (0.0259)	<0.001
Average hours per week of regularly scheduled lectures or conferencesa	1.1987 (0.4446)	0.008
Constant	60.4093 (3.5799)	<0.001

Notes:

^aDuring first year; adjusted R²=0.4483; number of observations =109.

Abbreviation: ABP, American Board of Pediatrics.

Discussion

In most past studies, program size was significantly associated with the board pass rates of many specialties, pediatrics included;^11,13–15 this was similarly found in the univariate analysis of the current study. However, the multivariate analysis revealed that faculty-to-position ratio, and not program size, was a significant independent predictor. Given that the inverse of faculty-to-position ratio is the number of resident positions per faculty member, meaning that it relates to program size, it is possible that program size is just a confounding factor. Furthermore, until now, there has been no information on how faculty-to-position ratio relates to the pass rate of the ABP certifying exam, although this same relationship was recently found for internal medicine residencies.¹⁶ Thus, as long as residency programs have a sufficient faculty-to-position ratio, their sizes should not matter. Conversely, increasing the size of program without increasing the number of faculties may have a negative effect on pass rate.

The finding that %USMD was a significant predictor of ABP exam pass rate is unsurprising. In the past, a study on board pass rates in general surgery found that highly competitive residency programs are more likely to attract similarly competitive residents.¹⁷ Moreover, competitive medical specialties usually have higher matching rates among US senior medical students from Doctor of Medicine programs,¹ which implies that such programs should have higher percentages of these students. Therefore, it follows that programs with a high %USMD will have more competitive residents with a stronger drive to pass the board exams, thus increasing the pass rate.

The last significant predictor in the multivariate model was average hr/wk of lectures. As shown in Table 2, the average hr/wk of lectures had a mean of 7.25 hr/wk and an SD of 1.92. Although one might infer from the SD that the differences among the programs were rather small, 1.92 hr/wk is still equivalent to 99.84 hours per year or 299.52 hours every 3 years. In other words, increasing 1 hr/wk of lecturing time is equivalent to 52 hours per year. This accords with the findings of a study of emergency medicine residents who were at-risk of failing, based on their in-training examination performances. The study found that individualized educational plans (eg, self-study audio review lectures) could improve these residents’ board pass rates.¹⁸ Thus, both statistically and intuitively, pass rates can be improved by increasing the amount of lectures or academic activities.

The vast majority of program characteristics were not independent predictors, including location, which was previously reported as significant predictor.¹¹ These discrepancies could have arisen from different statistical methods, variable types of characteristics and pass rate, population samples, or assumptions. It is worth noting that the data from FREIDA^® reported only the usage of in-training examination scores for evaluating the performance of residents, not the average scores in the program. For this reason, the current study could not measure the association between in-training examination scores and the pass rates. Specifically, I used multivariate linear regression, and treated board pass rate of residency programs as a continuous variable; other studies might have dichotomized the pass rate, used t-tests, or determined the correlations between individual pass rates and associated factors.

This study has several limitations. First, the data from FREIDA^® and the ABP website were not immune to human errors by data reporters or gatherers. Second, no programs that opted out of reporting program characteristics to FREIDA^® could be studied; whether programs opt out of reporting might be a predictor of pass rate. Third, I included only data on program characteristics collected at a single point in time, which does not allow me to infer whether these relationships change at different times. These limitations can be resolved by conducting further analyses with a larger dataset and with a longitudinal design. Furthermore, despite the limitations inherent to self-reported data, I believe the findings of this study can still be applied to most pediatrics residency programs in the US.

The results can also be used to improve pediatrics residency programs. First, programs should not focus on increasing program size, but rather on improving the level of supervision by balancing the faculty-to-position ratio. Second, programs should pay more attention to regular academic activities, such as lectures or conferences, as this would help pediatrics residents achieve better academic performance, thereby improving patients’ outcome.

Conclusion

Passing the ABP certifying exam relies on the competitiveness of individual residents and the quality of the training environment. In practice, these two factors are closely related. The fact that faculty-to-position ratio was a significant predictor highlights the benefits of a well-supervised training environment, whereas a higher %USMD indicates the necessity of greater competition in residency programs. Finally, a longer time spent on regular academic activities is associated with better academic outcomes, both statistically and intuitively.