ABSTRACT
BACKGROUND
Maintenance of certification examination performance is associated with quality of care. We aimed to examine relationships between electronic medical knowledge resource use, practice characteristics and examination scores among physicians recertifying in internal medicine.
METHODS
We conducted a cross-sectional study of 3,958 United States physicians who took the Internal Medicine Maintenance of Certification Examination (IM–MOCE) between January 1, 2006 and December 31, 2008, and who held individual licenses to one or both of two large electronic knowledge resource programs. We examined associations between physicians’ IM–MOCE scores and their days of electronic resource use, practice type (private practice, residency teaching clinic, inpatient, nursing home), practice model (single or multi-specialty), sex, age, and medical school location.
RESULTS
In the 365 days prior to the IM–MOCE, physicians used electronic resources on a mean (SD, range) of 20.3 (36.5, 0–265) days. In multivariate analyses, the number of days of resource use was independently associated with increased IM–MOCE scores (0.07-point increase per day of use, p = 0.02). Increased age was associated with decreased IM–MOCE scores (1.8-point decrease per year of age, p < 0.001). Relative to physicians working in private practice settings, physicians working in residency teaching clinics and hospital inpatient practices had higher IM–MOCE scores by 29.1 and 20.0 points, respectively (both p < 0.001).
CONCLUSIONS
Frequent use of electronic resources was associated with modestly enhanced IM–MOCE performance. Physicians involved in residency education clinics and hospital inpatient practices had higher IM–MOCE scores than physicians working in private practice settings.
KEY WORDS: Internal Medicine Maintenance of Certification Examination, IM–MOCE, scores, education
The purpose of certification of physicians is to ensure the safety of patients and the public by providing evidence of individual physician competency.1 In 1990, the American Board of Medical Specialties required all certification boards to issue time-limited certificates, thereby encouraging physicians to demonstrate maintenance of competency by renewing their certification at regular intervals throughout their careers. Although there is debate among physicians about the role and relevance of maintenance of certification (MOC),2 many practicing physicians believe that MOC is necessary,3 and some advocate that it is an important element of professionalism and public accountability.4 Patients value certification and many indicate that they would change physicians if they discovered that their doctor had failed to maintain certification.5
Maintenance of certification includes successfully passing a secure medical knowledge examination1; for internists this examination is the American Board of Internal Medicine Maintenance of Certification Examination (IM–MOCE).8 Substantial evidence for the validity of certification examinations has been demonstrated.5–7 Certification examination scores correlate with training factors (type of medical school and residency program)9,10 and assessments of competency by program directors,11 and recertification scores discriminate between individuals with and without subspecialty content expertise.12 A growing body of evidence demonstrates relationships between certification status and quality of care including outcomes following acute MI,13–15colorectal surgery,16 diabetes care,17 mammography screening,17 and other preventive services.18
Unfortunately, unrelenting time pressures in clinical practice have made it increasingly difficult for physicians to maintain habits of lifelong learning,19,20 resulting in decrements in medical knowledge over time.9,21,22 Traditional continuing medical education (CME) courses vary in effectiveness23–25 and often do not provide the “just in time” learning needed at the point of care.26 Critics of the MOC process question whether correlations between certification examination scores and quality of care measures represent true cause and effect or whether examination scores may be merely a surrogate marker of competency.27 They suggest that the ability to access relevant information resources to support decision-making is more important than fund of knowledge.27
However, research into clinical reasoning and cognition demonstrates that a robust fund of knowledge in working memory is necessary to recognize gaps in knowledge and when to use information resources for the benefit of patients.28,29 Since physicians need both a strong foundation of knowledge and the skills to effectively use information resources, it follows that regular and active use of electronic information resources could lead to gains in foundational knowledge. Electronic knowledge resources are capable of providing physicians quick access to evidence-based information, but the effectiveness of these resources in enhancing physicians’ knowledge is unknown. One study demonstrated a positive association between use of an electronic resource by residents and performance on a standardized examination30; however, this relationship has not been examined on a national scale.
The primary aim of this study was to examine relationships between electronic knowledge resource use and MOC examination scores among practicing internal medicine physicians in the United States. This study was designed to address current gaps in the literature by using a direct and objective measure of electronic resource use, a standardized, well-validated measure of medical knowledge (the IM–MOCE), and a large, national sample of physicians. A secondary aim of this study was to identify demographic and practice factors associated with IM–MOCE scores.
METHODS
Design, Setting and Participants
We conducted a cross-sectional study of 3,958 physicians in the United States who took the IM–MOCE for their first time between January 1, 2006 and December 31, 2008 and who held individual licenses to one or both of the electronic medical knowledge resources examined in the study. This study was approved by the Mayo Clinic Institutional Review Board.
Electronic Medical Knowledge Resources
This study examined usage of two leading electronic medical resources: UpToDate® and the American College of Physicians Physicians’ Information and Education Resource (PIER). UpToDate® and PIER are web-based resources that provide peer-reviewed, evidence-based information on a wide range of clinical topics, and are designed to answer clinical questions at the point-of-care.31,32 To maintain confidentiality of resource-specific data according to data access agreements with each resource, data were evaluated in aggregate and randomly labeled only as resource A and resource B.
We measured individual physicians’ use of either or both of the two electronic medical knowledge resources in the 365 days prior to taking the IM–MOCE. Resource accessions were electronically tracked for each physician by the date of accession. An accession was defined as logging on and opening up a resource regardless of the amount of time the resource remained open or the specific content viewed. Resource usage was recorded as the number of days in which one or more accessions occurred. These data were obtained from the resource companies in a de-indentified fashion and were collected in compliance with terms of use agreements for users of each resource.
Maintenance of Certification Examination Scores
The primary dependent variable was the physician’s IM–MOCE equated score on their first exam attempt. Equating is a measurement technique that enables scores from exams across administrations or forms to be used interchangeably so they can be directly compared. The score scale ranged from 200 to 800, with a mean of 500 and standard deviation of 100. To pass, physicians needed to correctly answer approximately 65% of the exam questions; this cutoff was set using an absolute standard through the well-established modified-Angoff method.33 This cut-score corresponds to a percentile of 18–22% for those taking the exam for the first time, depending on which form of the exam was taken. Considerable content and criterion validity evidence for IM–MOCE scores has been demonstrated.5
We also examined factors known or hypothesized to affect examination performance, including physicians’ age, gender, medical school location (United States or International), practice model (single or multi-specialty) and practice type (including private practice, academic medical center, hospital inpatient-based, residency clinic, and nursing home).
Data Collection and Analyses
Electronic resource usage data were linked with IM–MOCE score data for each physician using a matching algorithm combining physicians’ first and last names, birth dates, and postal addresses. We required a 100% match of characters for birth date and zip code to determine a positive match between resource usage data and examination scores for each physician. A de-identified data set was created for use in all analyses.
Standard univariate statistics were used to characterize the sample. Bivariate and multivariate analyses were conducted using generalized linear models, with IM–MOCE score as a continuous outcome. Independent variables (or sets of variables if linked, such as for licenses to electronic resources A, B, or both) were entered into the multivariate model if the univariate p < 0.20 and retained in a backward stepwise model selection procedure if p < 0.05. Two-way interaction terms and nonlinearity were assessed for all variables. Models were also evaluated for undue effects of outlier data. Statistical analyses were conducted using SAS Version 9.1 (SAS Institute Inc, Cary, NC).
Role of the Funding Source
This study received funding from the American Board of Internal Medicine Foundation (ABIMF). The co-authors affiliated with the ABIM (EH, RL, CJ) had a partial role in the design and conduct of the study, collection of the data, review and approval of the manuscript. No funding was received from UpToDate® or PIER resource companies.
RESULTS
Demographic Characteristics of Physicians
Of the 15,148 physicians who took the IM–MOCE between January 1, 2006 and December 31, 2008, 3,958 physicians were identified holding individual licenses to one or both electronic medical knowledge resources. The demographic characteristics of the 3,958 physicians with resource licenses were generally similar to the examinees without resources licenses, although those with licenses were somewhat younger (Table 1). The majority of physicians taking the IM–MOCE were graduates of United States medical schools, and most were between 40 and 49 years of age. Among physicians with resource licenses, approximately half (49.2%) reported working in a single-specialty practice. Nearly one-fifth (18.6%) were affiliated with an inpatient hospital practice, 15.6% were in an academic practice, and 6.6% worked in a residency clinic. Physicians with resource licenses were more likely to pass the IM–MOCE on the first attempt (88.9% vs. 80.8%, p < 0.001) and had a higher mean IM–MOCE score (555.8 vs. 537.5, p < 0.001).
Table 1.
Demographic Characteristics of Internal Medicine-Maintenance of Certification Examinees
| IM–MOCE Examinees with Electronic Resource Licenses (n = 3,958) | IM–MOCE Examinees without Electronic Resource Licenses (n = 11,190) | |||
|---|---|---|---|---|
| Variable | Number (%) | Mean IM–MOCE Score | Number (%) | Mean IM–MOCE Score |
| Gender | ||||
| Male | 2,508 (63.4) | 535.5 | 7,219 (64.5) | 504.7 |
| Female | 1,450 (36.6) | 533.8 | 3,971 (35.5) | 501.5 |
| Age | ||||
| <40 years | 1,161 (29.3) | 570.4 | 1,736 (15.5) | 548.9 |
| 40-49 years | 2,303 (58.2) | 528.2 | 7,346 (65.6) | 506.8 |
| ≥50 years | 494 (12.5) | 482.2 | 2,107 (18.8) | 454.8 |
| Missing | 1 (0.0) | |||
| United States Medical School | ||||
| Yes | 2,360 (59.6) | 544.7 | 5,961 (53.2) | 514.7 |
| No | 1595 (40.3) | 520.0 | 5,226 (46.7) | 490.9 |
| Missing | 3 (0.1) | 3 (0.0) | ||
| Practice Model | ||||
| Multi-specialty | 1,488 (37.6) | 545.4 | 3,426 (30.6) | 517.8 |
| Single-specialty | 1,948 (49.2) | 525.0 | 5,923 (52.9) | 495.4 |
| Not applicable | 331 (8.4) | 546.2 | 1,200 (10.7) | 505.2 |
| Missing | 191 (4.8) | 641 (5.7) | ||
| Practice Type | ||||
| Private practice | 1,810 (45.7) | 519.0 | 5,735 (51.3) | 489.8 |
| Academic | 617 (15.6) | 561.4 | 1,443 (12.9) | 539.3 |
| Residency clinic | 260 (6.6) | 564.2 | 494 (4.4) | 539.4 |
| Hospital inpatient | 736 (18.6) | 549.4 | 1,633 (14.6) | 517.9 |
| Nursing home | 108 (2.7) | 503.5 | 360 (3.2) | 484.1 |
| Other/Missing | 427 (10.8) | 1,525 (13.6) | ||
| IM–MOCE Pass on first attempt | ||||
| Yes | 3,517 (88.9) | 555.8 | 9,041 (80.8) | 537.5 |
| No | 441 (11.1) | 367.5 | 2,149 (19.2) | 360.8 |
IM–MOCE = Internal Medicine—Maintenance of Certification Examination
Associations with Maintenance of Certification Examination Scores
The mean (SD) equated score on the IM–MOCE among all 3,958 physicians with licenses to electronic resources was 534.9 (92.8). A total of 3,553 (89.8%) physicians had a license to only one of the two resources in this study, and 405 (10.2%) had a license to both resources. In the 365 days prior to the IM–MOCE, physicians with licenses to either or both resources used at least one resource on a mean (SD, range) of 20.3 (36.5, 0–265) days. Physicians with a license to only one resource used the resource on a mean of 18.3 (34.5, 0–264) days, while physicians with licenses to both resources used one or both of the resources on a mean of 38.4 (47.1, 0–265) days in the year prior to the recertification exam. Those with licenses to both resources were more likely to be United States medical graduates (68.6% vs. 58.7%, p < 0.001) and were more likely to report working primarily in a hospital inpatient practice (24.4% vs. 17.9%, p < 0.001). In addition, those with licenses to both resources had slightly higher mean initial certification examination scores (491.8 vs. 481.8, p = 0.04). A total of 1,193 (30.1%) physicians with licenses to electronic resources did not use the resources at all during the year. The mean IM–MOCE score for this group was slightly but not statistically significantly lower than that of the group with any use of the resources (531.7 vs. 536.2, p = 0.16).
Bivariate analyses are shown in Table 2. Total days of use of electronic resources was associated with a modest but statistically significant increase in mean IM–MOCE score of 0.14 points per day of use (p < 0.001). Physicians with licenses to both resources had higher mean IM–MOCE scores by 25.7 and 18.0 points compared to those with licenses to resource A and resource B, respectively (p < 0.001 and p < 0.001, respectively). The strongest explanatory associations as indicated by the R2 measure of explained variance occurred for age and prior internal medicine certification experience. IM–MOCE score decreased by 5.9 points for each year of age (p < 0.001). IM–MOCE score increased by 0.67 points for each point on the initial certification examination (p < 0.001), and decreased markedly as the number of attempts required to pass the initial certification examination increased (p < 0.001). Additional bivariate associations with IM–MOCE score were observed for medical school location and practice characteristics (Table 2).
Table 2.
Bivariate Associations with Maintenance of Certification Examination Scores among 3,958 Physicians
| Variable | Change in IM–MOCE Score* | 95% CI | p | R2 |
|---|---|---|---|---|
| Electronic Resources | ||||
| Total days of electronic resource use | 0.14 | 0.06, 0.22 | <0.001 | <0.01 |
| License to Resource A (vs. license to both) | −25.65 | −36.04, -15.25 | <0.001 | <0.01 |
| License to Resource B (vs. license to both) | −18.02 | −28.12, -7.92 | ||
| Demographics | ||||
| Age | −5.88 | −6.43, -5.33 | <0.001 | 0.11 |
| Male (vs. female) | 1.80 | −8.01, 4.40 | 0.57 | <0.01 |
| United States medical graduate (vs. international medical graduate) | 25.50 | 19.52, 31.49 | <0.001 | 0.02 |
| Certification Examination | ||||
| Initial certification score | 0.67 | 0.65, 0.69 | <0.001 | 0.46 |
| Number of attempts to pass initial certification examination (1, 2, 3, >3) | −55.86 | −59.71, -52.01 | <0.001 | 0.18 |
| Certification in subspecialty | −14.85 | −21.56, -8.14 | <0.001 | <0.01 |
| Single-specialist medical practice (vs. multi-specialist or other practice) | −20.78 | −26.69, -14.87 | <0.001 | 0.01 |
| Time spent in principal patient care role (%) | −0.39 | −0.54, -0.24 | <0.001 | <0.01 |
| Primary Practice Description (each vs. all others) | ||||
| Private practice | −30.76 | −36.63, -24.90 | <0.001 | 0.03 |
| Academic faculty practice | 31.80 | 23.84, 39.77 | <0.001 | 0.02 |
| Residency teaching clinic practice | 31.58 | 19.90, 43.26 | <0.001 | <0.01 |
| Hospital inpatient practice | 18.15 | 10.68, 25.62 | <0.001 | <0.01 |
| Nursing home/Long-term care facility practice | −32.24 | −50.01, -14.46 | <0.001 | <0.01 |
| Other practice | 3.86 | −3.26, 10.98 | 0.29 | <0.01 |
IM–MOCE = Internal Medicine—Maintenance of Certification Examination
*Change in IM–MOCE score for a 1-unit change in each independent variable
The multivariate model adjusting for demographic and practice characteristics accounted for 51.2% of the variability in the data. The number of days of use of electronic resources remained independently associated with an increase in IM–MOCE score, with an increase in IM–MOCE score of 0.07 points per day of use (p = 0.02) (Table 3). Thus, every one day of resource use within the 365 days prior to taking the IM–MOCE was associated with an examination score increase of 0.07 points (i.e., every 100 days of resource use was associated with an IM–MOCE score increase of 7 points). Days of resource use accounted for less than 1% of the variability in IM–MOCE score explained by the multivariate model.
Table 3.
Multivariate Analysis Examining Relationships between Electronic Resource Use, Physician Demographics, Practice Characteristics, and Maintenance of Certification Examination Scores among 3,958 Physicians
| Independent Variable | Change in IM–MOCE Score* | 95% CI | p |
|---|---|---|---|
| Electronic Resources | |||
| Total days of electronic resource use | 0.07 | 0.01, 0.13 | 0.02 |
| License to Resource A (vs. license to both) | −14.27 | −21.69, -6.85 | <0.001 |
| License to Resource B (vs. license to both) | −6.25 | −13.77, 1.27 | |
| Demographics | |||
| Age | −1.78 | −2.27, -1.30 | <0.001 |
| United States medical graduate (vs. international medical graduate) | 7.00 | 2.47, 11.54 | 0.003 |
| Certification Examination | |||
| Initial certification score | 0.87 | 0.81, 0.93 | <0.001 |
| Number of attempts to pass initial certification examination (1, 2, 3, >3) | 64.43 | 49.53, 79.33 | <0.001 |
| Certification in subspecialty | −10.99 | −16.03, -5.96 | <0.001 |
| Single-specialist Medical Practice (vs. multi-specialist or other practice) | −7.74 | −12.18, -3.29 | 0.003 |
| Primary Practice Description (each vs. all others) | |||
| Private practice | −14.11 | −18.87, -9.36 | <0.001 |
| Residency teaching clinic practice | 14.97 | 6.43, 23.50 | <0.001 |
| Hospital inpatient practice | 5.90 | 0.19, 11.61 | 0.03 |
| Nursing home/Long-term care facility practice | −14.47 | −27.40, -1.55 | 0.010 |
IM–MOCE = Internal Medicine—Maintenance of Certification Examination
Final model R2 = 0.51, intercept estimate = 222.71
*Change in IM–MOCE score for a 1-unit change in each independent variable
Initial score and fewer attempts to pass also remained strongly associated with higher IM–MOCE scores in the multivariate model. The remaining multivariate associations were also consistent with the bivariate results, although the effect estimates were generally attenuated. For example, each year of increased age was associated with a 1.8-point lower IM–MOCE score in the multivariate model (p < 0.001). Also, relative to physicians working in private practice settings, physicians working in residency teaching clinics and hospital inpatient practices had higher IM–MOCE scores by 29.1 and 20.0 points, respectively (both p < 0.001).
DISCUSSION
This study of a national sample of practicing physicians showed a positive association between the use of electronic medical knowledge resources and maintenance of certification examination scores, although the magnitude of this association was small. Additional associations were found between IM–MOCE score and prior certification experience, demographic factors, and practice characteristics, as discussed below.
The strongest predictor of IM–MOCE score examined in this study was the physician’s initial certification score, which by itself accounted for 46% of the variability in IM–MOCE score. This observation is consistent with prior studies demonstrating positive correlations between standardized test scores, including correlations between the Medical College Admissions Test (MCAT) and Steps 1, 2 and 3 of the United States Licensing Examination USMLE,34 between USMLE scores and the Internal Medicine In-Training Examination (ITE),35 and between ITE scores and the ABIM initial certification examination.36,37
In addition to the initial certification examination score, several notable associations were seen. First, this study showed a decrease in examination scores with increasing age. This finding is consistent with prior research showing declines in knowledge and clinical performance over time.9,21,22 Physicians face a tremendous challenge in keeping current with rapidly evolving medical advances over careers that may span several decades. Ongoing research into best practices for CME may help address this issue.22,24
Second, working primarily in a residency clinic was associated with a 15-point higher IM–MOCE score, which was equivalent in association to approximately 217 days of electronic resource use. Although further study is needed, this novel finding may suggest that interacting with residents in outpatient clinics may provide valuable intellectual stimulation for the supervising physician and assist physicians in keeping their knowledge and clinical practice current. Studies of the behaviors of clinical supervisors in residency clinics show that they frequently use internet-based resources to answer questions raised during the interaction with learners38 and role model positive behaviors for asking and seeking answers for clinical questions.39 These behaviors may contribute to more effective maintenance of current medical knowledge. In addition, faculty likely participate in a number of other educational activities in the program such as morning report, grand rounds, teaching rounds and curriculum development.
Third, physicians in multi-specialty practices had higher IM–MOCE scores compared to those in single-specialty practices, as did those with primarily inpatient hospital practices as opposed to solo private practices. It is unknown whether these differences are attributable to better access to traditional CME activities or to greater opportunities for more proximal interactions with colleagues with diverse expertise. Perhaps being a member of a multifaceted “learning community”, as exists in most multi-specialty practices, hospitals, and residency clinics, offers an advantage in terms of maintaining one’s knowledge base.40
There are several limitations to this study. First, this is a cross sectional study. Causal relationships between electronic resource use and examination scores cannot be assumed from these data. Second, we were able to examine use of just two electronic resources. Although PIER and UpToDate® are two very large resources with over 400,000 users combined,31,32 this study cannot account for use of other electronic resources or for general internet searching. Third, only physicians with individual licenses to these electronic resources were included. It is noteworthy that resource users scored higher on the certification examination than non-users, independent of the amount of resource use and after including other explanatory variables in the multivariate models. A possible explanation for this finding is that physicians who access information resources may be more likely than others to engage in the broader process of evidence-based practice which involves active learning at the point of care, and may result in greater knowledge acquisition and retention.29 However, this study measured only resource access and examination scores; further research is needed to examine relationships between other aspects of evidence-based practice and medical knowledge. Fourth, resource usage was directly measured through usage logs identified by individual accounts; therefore the analyses do not consider potential access by participants to these resources through institutional licenses. Fifth, electronic resources may differ in the extent to which they “push” or “pull” evidence to clinicians, meaning some resources provide general information to clinicians through online and print media while other resources require physicians to actively acquire specific information when needed.41 We were unable to explore such differences in this study and this remains an important area for future research. Finally, although participants in this study had higher IM–MOCE scores on average, it is reassuring that the score patterns across available demographic factors were very similar for participants and the remainder of the IM–MOCE examinees. However, the observed associations between physician demographics, practice characteristics, and IM–MOCE scores should be confirmed for other physicians participating in MOC programs.
This study also has several strengths. It examined a large, national sample of physicians from a variety of practice settings recertifying in internal medicine over a 3-year period. Additionally, a substantial body of validity evidence supports the primary outcome, the IM–MOCE score, which is a well-established and credible measure of medical knowledge. Finally, the primary independent variable, daily electronic medical knowledge resource use, was measured directly and objectively.
In summary, the results of this study indicate that regular, frequent use of electronic medical knowledge resources may be associated with modestly enhanced performance on the Internal Medicine Maintenance of Certification Examination, while relatively larger score increases are associated with involvement in multi-specialty practice and active participation in a residency education clinic. The factors underlying these associations are worthy of further study.
Acknowledgments
Contributors
The authors wish to thank Peter A. Bonis, MD of UpToDate for his assistance in providing UpToDate® usage data and Gary Glenn for his assistance in providing PIER usage data.
Funding
This study received funding from the American Board of Internal Medicine Foundation (ABIMF). This organization had a partial role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. Specifically, Dr. Holmboe, Dr. Lipner, and Ms. Jacobs, each of whom are affiliated with the ABIM, reviewed the study protocol, provided ABIM administrative data, and reviewed and approved the final manuscript. This study was also supported in part by the Mayo Clinic Internal Medicine Residency Office of Educational Innovations as part of the ACGME Educational Innovations Project. No funding was received from UpToDate® or PIER resource companies. The electronic medical knowledge resource companies examined in this study, UpToDate® and PIER, had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.
Prior presentations
This study was presented at the national meeting of the Society of General Internal Medicine in May, 2011.
Conflict of Interest
The authors declare they do not have a conflict of interest. Dr. Holmboe, Dr. Lipner, and Ms. Jacobs, each of whom are affiliated with the ABIM, reviewed the study protocol, provided ABIM administrative data, and reviewed and approved the final manuscript.
Footnotes
Funding
This study received funding from the American Board of Internal Medicine Foundation (ABIMF). Please see the Acknowledgments for the role of the funding source and conflicts of interest.
References
- 1.American Board of Medical Specialties. Maintenance of Certification. http://www.abms.org/Maintenance_of_Certification/ABMS_MOC.aspx. Accessed January 7, 2012.
- 2.Drazen JM, Weinstein DF. Considering recertification. N Engl J Med. 2010;362:946–947. doi: 10.1056/NEJMe1000174. [DOI] [PubMed] [Google Scholar]
- 3.Lipner RS, Bylsma WH, Arnold GK, Fortna GS, Tooker J, Cassel CK. Who is maintaining certification in internal medicine–and why? A national survey 10 years after initial certification. Ann Intern Med. 2006;144(1):29–36. doi: 10.7326/0003-4819-144-1-200601030-00007. [DOI] [PubMed] [Google Scholar]
- 4.Cassel CK, Holmboe ES. Credentialing and public accountability: a central role for board certification. JAMA. 2006;295(8):939–940. doi: 10.1001/jama.295.8.939. [DOI] [PubMed] [Google Scholar]
- 5.Brennan TA, Horwitz RI, Duffy FD, Cassel CK, Goode LD, Lipner RS. The role of physician specialty board certification status in the quality movement. JAMA. 2004;292(9):1038–1043. doi: 10.1001/jama.292.9.1038. [DOI] [PubMed] [Google Scholar]
- 6.Sharp LK, Wang R, Lipsky MS. Perception of Competency to Perform Procedures and Future Practice Intent: A National Survey of Family Practice Residents. Acad Med. 2003;78(9):926–932. doi: 10.1097/00001888-200309000-00019. [DOI] [PubMed] [Google Scholar]
- 7.Holmboe ES, Lipner R, Greiner A. Assessing quality of care: knowledge matters. JAMA. 2008;299(3):338–340. doi: 10.1001/jama.299.3.338. [DOI] [PubMed] [Google Scholar]
- 8.American Board of Internal Medicine. Requirements for Maintenance of Certification. http://www.abim.org/moc/requirements-for-MOC.aspx. Accessed January 7, 2012.
- 9.Norcini JJ, Shea JA, Benson JA., Jr Changes in the medical knowledge of candidates for certification. Ann Intern Med. 1991;114(1):33–35. doi: 10.7326/0003-4819-114-1-33. [DOI] [PubMed] [Google Scholar]
- 10.Norcini JJ, Grosso LJ, Shea JA, Webster GD. The relationship between features of residency training and ABIM certifying examination performance. J Gen Intern Med. 1987;2(5):330–336. doi: 10.1007/BF02596169. [DOI] [PubMed] [Google Scholar]
- 11.Norcini JJ, Webster GD, Grosso LJ, Blank LL, Benson JA., Jr Ratings of residents’ clinical competence and performance on certification examination. J Med Educ. 1987;62(6):457–462. doi: 10.1097/00001888-198706000-00001. [DOI] [PubMed] [Google Scholar]
- 12.Norcini JJ, Lipner RS, Benson JA, Jr, Webster GD. An analysis of the knowledge base of practicing internists as measured by the 1980 recertification examination. Ann Intern Med. 1985;102(3):385–389. doi: 10.7326/0003-4819-102-3-385. [DOI] [PubMed] [Google Scholar]
- 13.Norcini JJ, Lipner RS, Kimball HR. Certifying examination performance and patient outcomes following acute myocardial infarction. Med Educ. 2002;36(9):853–859. doi: 10.1046/j.1365-2923.2002.01293.x. [DOI] [PubMed] [Google Scholar]
- 14.Norcini JJ, Boulet JR, Dauphinee WD, Opalek A, Krantz ID, Anderson ST. Evaluating the quality of care provided by graduates of international medical schools. Health Affairs. 2010;29(8):1461–1468. doi: 10.1377/hlthaff.2009.0222. [DOI] [PubMed] [Google Scholar]
- 15.Chen J, Rathore SS, Wang Y, Radford MJ, Krumholz HM. Physician board certification and the care and outcomes of elderly patients with acute myocardial infarction. J Gen Intern Med. 2006;21(3):238–244. doi: 10.1111/j.1525-1497.2006.00326.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Prystowsky JB, Bordage G, Feinglass JM. Patient outcomes for segmental colon resection according to surgeon’s training, certification, and experience. Surgery. 2002;132(4):663–670. doi: 10.1067/msy.2002.127550. [DOI] [PubMed] [Google Scholar]
- 17.Holmboe ES, Wang Y, Meehan TP, et al. Association between maintenance of certification examination scores and quality of care for Medicare beneficiaries. Arch Intern Med. 2008;168(13):1396–1403. doi: 10.1001/archinte.168.13.1396. [DOI] [PubMed] [Google Scholar]
- 18.Pham HH, Schrag D, Hargraves JL, Bach PB. Delivery of preventive services to older adults by primary care physicians. JAMA. 2005;294(4):473–481. doi: 10.1001/jama.294.4.473. [DOI] [PubMed] [Google Scholar]
- 19.Davidoff F. Continuing Medical Education Resources. J Gen Intern Med. 1997;12(S2):S15–S19. doi: 10.1046/j.1525-1497.12.s2.2.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Duffy FD, Holmboe ES. Self-assessment in lifelong learning and improving performance in practice: physician know thyself. JAMA. 2006;296(9):1137–1139. doi: 10.1001/jama.296.9.1137. [DOI] [PubMed] [Google Scholar]
- 21.Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142(4):260–273. doi: 10.7326/0003-4819-142-4-200502150-00008. [DOI] [PubMed] [Google Scholar]
- 22.Norcini JJ, Maihoff NA, Day SC, Benson JA., Jr Trends in medical knowledge as assessed by the certifying examination in internal medicine. JAMA. 1989;262(17):2402–2404. doi: 10.1001/jama.1989.03430170064029. [DOI] [PubMed] [Google Scholar]
- 23.Marinopoulos SS, Dorman T, Ratanawongsa N, et al. Effectiveness of continuing medical education. Evid Rep Tech Assess (Full Rep) 2007;149:1–69. [PMC free article] [PubMed] [Google Scholar]
- 24.Bloom BS. Effects of continuing medical education on improving physician clinical care and patient health: a review of systematic reviews. Int J Technol Assess Health Care. 2005;21(3):380–385. doi: 10.1017/S026646230505049X. [DOI] [PubMed] [Google Scholar]
- 25.Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA. 1995;274(9):700–705. doi: 10.1001/jama.1995.03530090032018. [DOI] [PubMed] [Google Scholar]
- 26.Leach DC, Fletcher SW. Perspectives on continuing education in the health professions: improving health care through lifelong learning. Chest. 2008;134(6):1299–1303. doi: 10.1378/chest.08-1022. [DOI] [PubMed] [Google Scholar]
- 27.Goldman L, Goroll AH, Kessler B. Clinical Decisions. American Board of Internal Medicine Maintenance of Certification Program, Recommendation 2: Do not enroll in the current MOC program. N Engl J Med. 2010;362:950–952. doi: 10.1056/NEJMclde0911205. [DOI] [PubMed] [Google Scholar]
- 28.Gruppen LD, Frohna AZ. Clinical reasoning. In: Norman GR, Vleuten CP, Newble DI, editors. International Handbook of Research in Medical Education. Dordrecht: Kluwer Academic; 2002. pp. 205–230. [Google Scholar]
- 29.Bowen JL. Educational strategies to promote clinical diagnostic reasoning. N Engl J Med. 2006;355:2217–2225. doi: 10.1056/NEJMra054782. [DOI] [PubMed] [Google Scholar]
- 30.McDonald FS, Zeger SL, Kolars JC. Factors associated with medical knowledge acquisition during internal medicine residency. J Gen Intern Med. 2007;22(7):962–968. doi: 10.1007/s11606-007-0206-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.American College of Physicians. PIER, Physicians’ information and Education Resource. Available at: http://pier.acponline.org/overview.html. Accessed January 7, 2012.
- 32.UpToDate. About the company. Available at: http://www.uptodate.com/home/about/about.html. Accessed January 7, 2012.
- 33.Brennan RL, Ed. Educational Measurement. Fourth ed. Westport, CT: Praeger Publishers; 2006.
- 34.Callahan CA, Hojat M, Veloski J, Erdmann JB, Gonnella JS. The Predictive Validity of Three Versions of the MCAT in Relation to Performance in Medical School, Residency, and Licensing Examinations: A Longitudinal Study of 36 Classes of Jefferson Medical College. Acad Med. 2010;85(6):980–7. [DOI] [PubMed]
- 35.McDonald FS, Zeger SL, Kolars JC. Associations between United States Medical Licensing Examination (USMLE) and Internal Medicine In-Training Examination (IM-ITE) scores. J Gen Intern Med. 2008;23(7):1016–1019. doi: 10.1007/s11606-008-0641-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Grossman RS, Fincher RM, Layne RD, Seelig CB, Berkowitz LR, Levine MA. Validity of the in-training examination for predicting American Board of Internal Medicine certifying examination scores. J Gen Intern Med. 1992;7(1):63–67. doi: 10.1007/BF02599105. [DOI] [PubMed] [Google Scholar]
- 37.Babbott SF, Beasley BW, Hinchey KT, Blotzer JW, Holmboe ES. The predictive validity of the internal medicine in-training examination. Am J Med. 2007;120(8):735–740. doi: 10.1016/j.amjmed.2007.05.003. [DOI] [PubMed] [Google Scholar]
- 38.Carney PA, Poor DA, Schifferdecker KE, Gephart DS, Brooks WB, Nierenberg DW. Computer use among community-based primary care physician preceptors. Acad Med. 2004;79(6):580–590. doi: 10.1097/00001888-200406000-00017. [DOI] [PubMed] [Google Scholar]
- 39.McGee SR, Irby DM. Teaching in the Outpatient Clinic: Practical Tips. J Gen Intern Med. 1997;12(S2):S34–S40. doi: 10.1046/j.1525-1497.12.s2.5.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Lipner R, Song H, Biester T, Rhodes R. Factors that influence general internists’ and surgeons’ performance on maintenance of certification exams. Acad Med. 2010;86(1):53–58. doi: 10.1097/ACM.0b013e3181ff8283. [DOI] [PubMed] [Google Scholar]
- 41.Lavis JN, Lomas J, Hamid M, Sewankambo NK. Bull World Health Organ. 2006;84:620–628. doi: 10.2471/BLT.06.030312. [DOI] [PMC free article] [PubMed] [Google Scholar]